|---|
| [ View source on GitHub](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L50-L310) |
Composite `FeatureConnector`; each feature in `dict` has its own connector.
Inherits From: [`FeatureConnector`](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/FeatureConnector)
tfds.features.FeaturesDict(
feature_dict: Mapping[str, feature_lib.FeatureConnectorArg],
*,
doc: feature_lib.DocArg = None
)
The encode/decode method of the spec feature will recursively encode/decode
every sub-connector given on the constructor.
Other features can inherit from this class and call super() in order to get
nested container.
#### Example:
#### For DatasetInfo:
features = tfds.features.FeaturesDict({
'input': tfds.features.Image(),
'output': np.int32,
})
#### At generation time:
for image, label in generate_examples:
yield {
'input': image,
'output': label
}
At tf.data.Dataset() time:
for example in tfds.load(...):
tf_input = example['input']
tf_output = example['output']
For nested features, the FeaturesDict will internally flatten the keys for the
features and the conversion to tf.train.Example. Indeed, the tf.train.Example
proto do not support nested features while tf.data.Dataset does.
But internal transformation should be invisible to the user.
#### Example:
tfds.features.FeaturesDict({
'input': np.int32,
'target': {
'height': np.int32,
'width': np.int32,
},
})
Will internally store the data as:
{
'input': tf.io.FixedLenFeature(shape=(), dtype=tf.int32),
'target/height': tf.io.FixedLenFeature(shape=(), dtype=tf.int32),
'target/width': tf.io.FixedLenFeature(shape=(), dtype=tf.int32),
}
| ## Args ||
|---|---|
| `feature_dict` | Mapping containing the feature connectors of a example. The keys should correspond to the data dict as returned by tf.data.Dataset(). Types (np.int32,...) and dicts will automatically be converted into FeatureConnector. |
| `doc` | Documentation of this feature (e.g. description). |
| ## Raises ||
|---|---|
| `ValueError` | If one of the given features is not recognized |
| ## Attributes ||
|---|---|
| `doc` |
|
| `dtype` | Return the dtype (or dict of dtype) of this FeatureConnector. |
| `flat_features` |
|
| `flat_sequence_ranks` |
|
| `flat_serialized_info` |
|
| `np_dtype` |
|
| `numpy_dtype` |
|
| `shape` | Return the shape (or dict of shape) of this FeatureConnector. |
| `tf_dtype` |
|
| `tf_example_spec` | Returns the `tf.Example` proto structure. |
## Methods
### `catalog_documentation`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L174-L193)
catalog_documentation() -> list[feature_lib.CatalogFeatureDocumentation]
Returns the feature documentation to be shown in the catalog.
### `cls_from_name`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L401-L428)
@classmethod
cls_from_name(
python_class_name: str
) -> Type['FeatureConnector']
Returns the feature class for the given Python class.
### `decode_batch_example`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L780-L830)
decode_batch_example(
tfexample_data
)
Decode multiple features batched in a single tf.Tensor.
This function is used to decode features wrapped in
[`tfds.features.Sequence()`](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/Sequence).
By default, this function apply `decode_example` on each individual
elements using [`tf.map_fn`](https://www.tensorflow.org/api_docs/python/tf/map_fn). However, for optimization, features can
overwrite this method to apply a custom batch decoding.
| Args ||
|---|---|
| `tfexample_data` | Same [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) inputs as `decode_example`, but with and additional first dimension for the sequence length. |
| Returns ||
|---|---|
| `tensor_data` | Tensor or dictionary of tensor, output of the tf.data.Dataset object |
### `decode_batch_example_np`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L832-L834)
decode_batch_example_np(
batch
)
See decode_batch_example.
### `decode_example`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/top_level_feature.py#L106-L109)
decode_example(
serialized_example, *, decoders=None
)
Decode the feature dict to TF compatible input.
> [!NOTE]
> **Note:** If eager is not enabled, this function will be executed as a tensorflow graph (in [`tf.data.Dataset.map(features.decode_example)`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map)).
| Args ||
|---|---|
| `tfexample_data` | Data or dictionary of data, as read by the tf-example reader. It correspond to the [`tf.Tensor()`](https://www.tensorflow.org/api_docs/python/tf/Tensor) (or dict of [`tf.Tensor()`](https://www.tensorflow.org/api_docs/python/tf/Tensor)) extracted from the [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example), matching the info defined in `get_serialized_info()`. |
| Returns ||
|---|---|
| `tensor_data` | Tensor or dictionary of tensor, output of the tf.data.Dataset object |
### `decode_example_np`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/top_level_feature.py#L111-L114)
decode_example_np(
serialized_example, *, decoders=None
)
Decode the example data into NumPy-compatible input.
| Args ||
|---|---|
| `example_data` | Value to convert to NumPy. |
| Returns ||
|---|---|
| `np_data` | Data as NumPy-compatible type: either a Python primitive (bytes, int, etc) or a NumPy array. |
### `decode_ragged_example`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L836-L853)
decode_ragged_example(
tfexample_data
)
Decode nested features from a tf.RaggedTensor.
This function is used to decode features wrapped in nested
[`tfds.features.Sequence()`](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/Sequence).
By default, this function apply `decode_batch_example` on the flat values
of the ragged tensor. For optimization, features can
overwrite this method to apply a custom batch decoding.
| Args ||
|---|---|
| `tfexample_data` | [`tf.RaggedTensor`](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor) inputs containing the nested encoded examples. |
| Returns ||
|---|---|
| `tensor_data` | The decoded [`tf.RaggedTensor`](https://www.tensorflow.org/api_docs/python/tf/RaggedTensor) or dictionary of tensor, output of the tf.data.Dataset object |
### `deserialize_example`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/top_level_feature.py#L130-L150)
deserialize_example(
serialized_example: (tf.Tensor | bytes), *, decoders=None
) -> utils.TensorDict
Decodes the [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) data into [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor).
See `serialize_example` to encode the data into proto.
| Args ||
|---|---|
| `serialized_example` | The tensor-like object containing the serialized [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) proto. |
| `decoders` | Eventual decoders to apply (see [documentation](https://www.tensorflow.org/datasets/decode)) |
| Returns ||
|---|---|
| The decoded features tensors. ||
### `deserialize_example_np`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/top_level_feature.py#L152-L159)
deserialize_example_np(
serialized_example: (tf.Tensor | bytes | memoryview), *, decoders=None
) -> utils.NpArrayOrScalarDict
### `encode_example`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L234-L246)
encode_example(
example_dict
)
See base class for details.
### `from_config`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L646-L665)
@classmethod
from_config(
root_dir: str
) -> FeatureConnector
Reconstructs the FeatureConnector from the config file.
#### Usage:
features = FeatureConnector.from_config('path/to/dir')
| Args ||
|---|---|
| `root_dir` | Directory containing the features.json file. |
| Returns ||
|---|---|
| The reconstructed feature instance. ||
### `from_json`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L434-L480)
@classmethod
from_json(
value: Json
) -> FeatureConnector
FeatureConnector factory.
This function should be called from the [`tfds.features.FeatureConnector`](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/FeatureConnector)
base class. Subclass should implement the `from_json_content`.
#### Example:
feature = tfds.features.FeatureConnector.from_json(
{'type': 'Image', 'content': {'shape': [32, 32, 3], 'dtype': 'uint8'} }
)
assert isinstance(feature, tfds.features.Image)
| Args ||
|---|---|
| `value` | `dict(type=, content=)` containing the feature to restore. Match dict returned by `to_json`. |
| Returns ||
|---|---|
| The reconstructed FeatureConnector. ||
### `from_json_content`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L211-L224)
@classmethod
from_json_content(
value: (Json | feature_pb2.FeaturesDict)
) -> 'FeaturesDict'
FeatureConnector factory (to overwrite).
Subclasses should overwrite this method. This method is used when
importing the feature connector from the config.
This function should not be called directly. [`FeatureConnector.from_json`](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/FeatureConnector#from_json)
should be called instead.
See existing FeatureConnectors for implementation examples.
| Args ||
|---|---|
| `value` | FeatureConnector information represented as either Json or a Feature proto. The content must match what is returned by `to_json_content`. |
| `doc` | Documentation of this feature (e.g. description). |
| Returns ||
|---|---|
| The reconstructed FeatureConnector. ||
### `from_proto`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L618-L632)
@classmethod
from_proto(
feature_proto: feature_pb2.Feature
) -> T
Instantiates a feature from its proto representation.
### `get_serialized_info`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L203-L209)
get_serialized_info()
See base class for details.
### `get_tensor_info`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L195-L201)
get_tensor_info()
See base class for details.
### `get_tensor_spec`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L361-L370)
get_tensor_spec() -> TreeDict[tf.TensorSpec]
Returns the tf.TensorSpec of this feature (not the element spec!).
Note that the output of this method may not correspond to the element spec
of the dataset. For example, currently this method does not support
RaggedTensorSpec.
### `items`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L142-L143)
items()
### `keys`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L139-L140)
keys()
### `load_metadata`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L296-L310)
load_metadata(
data_dir, feature_name=None
)
See base class for details.
### `repr_html`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L855-L857)
repr_html(
ex: np.ndarray
) -> str
Returns the HTML str representation of the object.
### `repr_html_batch`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L859-L874)
repr_html_batch(
ex: np.ndarray
) -> str
Returns the HTML str representation of the object (Sequence).
### `repr_html_ragged`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L876-L878)
repr_html_ragged(
ex: np.ndarray
) -> str
Returns the HTML str representation of the object (Nested sequence).
### `save_config`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L634-L644)
save_config(
root_dir: str
) -> None
Exports the `FeatureConnector` to a file.
| Args ||
|---|---|
| `root_dir` | `path/to/dir` containing the `features.json` |
### `save_metadata`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L287-L294)
save_metadata(
data_dir, feature_name=None
)
See base class for details.
### `serialize_example`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/top_level_feature.py#L116-L128)
serialize_example(
example_data
) -> bytes
Encodes nested data values into [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example) bytes.
See `deserialize_example` to decode the proto into [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor).
| Args ||
|---|---|
| `example_data` | Example data to encode (numpy-like nested dict) |
| Returns ||
|---|---|
| The serialized [`tf.train.Example`](https://www.tensorflow.org/api_docs/python/tf/train/Example). ||
### `to_json`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L482-L545)
to_json() -> Json
Exports the FeatureConnector to Json.
Each feature is serialized as a `dict(type=..., content=...)`.
- `type`: The cannonical name of the feature (`module.FeatureName`).
- `content`: is specific to each feature connector and defined in `to_json_content`. Can contain nested sub-features (like for [`tfds.features.FeaturesDict`](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/FeaturesDict) and [`tfds.features.Sequence`](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/Sequence)).
#### For example:
tfds.features.FeaturesDict({
'input': tfds.features.Image(),
'target': tfds.features.ClassLabel(num_classes=10),
})
#### Is serialized as:
{
"type": "tensorflow_datasets.core.features.features_dict.FeaturesDict",
"content": {
"input": {
"type": "tensorflow_datasets.core.features.image_feature.Image",
"content": {
"shape": [null, null, 3],
"dtype": "uint8",
"encoding_format": "png"
}
},
"target": {
"type":
"tensorflow_datasets.core.features.class_label_feature.ClassLabel",
"content": {
"num_classes": 10
}
}
}
}
| Returns ||
|---|---|
| A `dict(type=, content=)`. Will be forwarded to `from_json` when reconstructing the feature. ||
### `to_json_content`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L226-L232)
to_json_content() -> feature_pb2.FeaturesDict
FeatureConnector factory (to overwrite).
This function should be overwritten by the subclass to allow re-importing
the feature connector from the config. See existing FeatureConnector for
example of implementation.
| Returns ||
|---|---|
| The FeatureConnector metadata in either a dict, or a Feature proto. This output is used in `from_json_content` when reconstructing the feature. ||
### `to_proto`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/feature.py#L589-L616)
to_proto() -> feature_pb2.Feature
Exports the FeatureConnector to the Feature proto.
For features that have a specific schema defined in a proto, this
function needs to be overriden. If there's no specific proto schema,
then the feature will be represented using JSON.
| Returns ||
|---|---|
| The feature proto describing this feature. ||
### `values`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L145-L146)
values()
### `__contains__`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L148-L149)
__contains__(
k
)
### `__getitem__`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L151-L153)
__getitem__(
key
)
Return the feature associated with the key.
### `__iter__`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L158-L159)
__iter__()
### `__len__`
[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/features/features_dict.py#L155-L156)
__len__()
| ## Class Variables ||
|---|---|
| ALIASES | `[]` |