tfds.features.Video

View source on GitHub

FeatureConnector for videos, encoding frames individually on disk.

Inherits From: Sequence, FeatureConnector

tfds.features.Video(
    shape: Sequence[int],
    encoding_format: str = 'png',
    ffmpeg_extra_args: Sequence[str] = (),
    use_colormap: bool = False,
    dtype=tf.uint8
)

Video: The image connector accepts as input a 4 dimensional tf.uint8 array representing a video, a sequence of paths to encoded frames, or a path or a file object that can be decoded with ffmpeg. Note that not all formats in ffmpeg support reading from pipes, so providing a file object might fail. Furthermore, if a path is given that is not on the local file system, we first copy it to a temporary local file before passing it to ffmpeg.

Output:

• video: tf.Tensor of type tf.uint8 and shape [num_frames, height, width, channels], where channels must be 1 or 3

Example:

• In the DatasetInfo object:

features=features.FeatureDict({
    'video': features.Video(shape=(None, 64, 64, 3)),
})

• During generation, you can use any of:

yield {
    'video': np.ones(shape=(128, 64, 64, 3), dtype=np.uint8),
}

or list of frames:

yield {
    'video': ['path/to/frame001.png', 'path/to/frame002.png'],
}

or path to video (including os.PathLike objects):

yield {
    'video': '/path/to/video.avi',
}

or file object (or bytes):

yield {
    'video': tf.io.gfile.GFile('/complex/path/video.avi'),
}

Args
shape	tuple of ints, the shape of the video (num_frames, height, width, channels), where channels is 1 or 3.
encoding_format	The video is stored as a sequence of encoded images. You can use any encoding format supported by image_feature.Feature.
ffmpeg_extra_args	A sequence of additional args to be passed to the ffmpeg binary. Specifically, ffmpeg will be called as: ffmpeg -i <input_file> <ffmpeg_extra_args> %010d.<encoding_format>
use_colormap	Forwarded to tfds.features.Image. If True, tfds.as_dataframe will display each value in the image with a different color.
dtype	tf.uint16 or tf.uint8 (default). tf.uint16 can be used only with png encoding_format

Raises
ValueError	If the shape is invalid

Attributes
dtype	Return the dtype (or dict of dtype) of this FeatureConnector.
feature	The inner feature.
shape	Return the shape (or dict of shape) of this FeatureConnector.

Methods

decode_batch_example

View source

decode_batch_example(
    tfexample_data
)

Decode multiple features batched in a single tf.Tensor.

This function is used to decode features wrapped in tfds.features.Sequence(). By default, this function apply decode_example on each individual elements using tf.map_fn. However, for optimization, features can overwrite this method to apply a custom batch decoding.

Args
tfexample_data	Same tf.Tensor inputs as decode_example, but with and additional first dimension for the sequence length.

Returns
tensor_data	Tensor or dictionary of tensor, output of the tf.data.Dataset object

decode_example

View source

decode_example(
    serialized_example, decoders=None
)

Decode the serialize examples.

Args
serialized_example	Nested dict of tf.Tensor
decoders	Nested dict of Decoder objects which allow to customize the decoding. The structure should match the feature structure, but only customized feature keys need to be present. See the guide for more info.

Returns
example	Nested dict containing the decoded nested examples.

decode_ragged_example

View source

decode_ragged_example(
    tfexample_data
)

Decode nested features from a tf.RaggedTensor.

This function is used to decode features wrapped in nested tfds.features.Sequence(). By default, this function apply decode_batch_example on the flat values of the ragged tensor. For optimization, features can overwrite this method to apply a custom batch decoding.

Args
tfexample_data	tf.RaggedTensor inputs containing the nested encoded examples.

Returns
tensor_data	The decoded tf.RaggedTensor or dictionary of tensor, output of the tf.data.Dataset object

encode_example

View source

encode_example(
    video_or_path_or_fobj
)

Converts the given image into a dict convertible to tf example.

from_config

View source

@classmethod
from_config(
    root_dir: str
) -> "FeatureConnector"

Reconstructs the FeatureConnector from the config file.

Usage:

features = FeatureConnector.from_config('path/to/features.json')

Args
root_dir	Directory containing to the features.json file.

Returns
The reconstructed feature instance.

from_json

View source

@classmethod
from_json(
    value: tfds.typing.Json
) -> "FeatureConnector"

FeatureConnector factory.

This function should be called from the tfds.features.FeatureConnector base class. Subclass should implement the from_json_content.

Example:

feature = tfds.features.FeatureConnector.from_json(
    {'type': 'Image', 'content': {'shape': [32, 32, 3], 'dtype': 'uint8'} }
)
assert isinstance(feature, tfds.features.Image)

Args
value	dict(type=, content=) containing the feature to restore. Match dict returned by to_json.

Returns
The reconstructed FeatureConnector.

from_json_content

View source

@classmethod
from_json_content(
    value: tfds.typing.Json
) -> "Video"

FeatureConnector factory (to overwrite).

Subclasses should overwritte this method. importing the feature connector from the config.

This function should not be called directly. FeatureConnector.from_json should be called instead.

This function See existing FeatureConnector for example of implementation.

Args
value	FeatureConnector information. Match the dict returned by to_json_content.

Returns
The reconstructed FeatureConnector.

get_serialized_info

View source

get_serialized_info()

See base class for details.

get_tensor_info

View source

get_tensor_info()

See base class for details.

load_metadata

View source

load_metadata(
    *args, **kwargs
)

See base class for details.

repr_html

View source

repr_html(
    ex: np.ndarray
) -> str

Video are displayed as <video>.

repr_html_batch

View source

repr_html_batch(
    ex: np.ndarray
) -> str

Returns the HTML str representation of the object (Sequence).

repr_html_ragged

View source

repr_html_ragged(
    ex: np.ndarray
) -> str

Returns the HTML str representation of the object (Nested sequence).

save_config

View source

save_config(
    root_dir: str
) -> None

Exports the FeatureConnector to a file.

Args
root_dir	path/to/dir containing the features.json

save_metadata

View source

save_metadata(
    *args, **kwargs
)

See base class for details.

to_json

View source

to_json() -> tfds.typing.Json

Exports the FeatureConnector to Json.

Each feature is serialized as a dict(type=..., content=...).

• type: The cannonical name of the feature (module.FeatureName).
• content: is specific to each feature connector and defined in to_json_content. Can contain nested sub-features (like for tfds.features.FeaturesDict and tfds.features.Sequence).

For example:

tfds.features.FeaturesDict({
    'input': tfds.features.Image(),
    'target': tfds.features.ClassLabel(num_classes=10),
})

Is serialized as:

{
    "type": "tensorflow_datasets.core.features.features_dict.FeaturesDict",
    "content": {
        "input": {
            "type": "tensorflow_datasets.core.features.image_feature.Image",
            "content": {
                "shape": [null, null, 3],
                "dtype": "uint8",
                "encoding_format": "png"
            }
        },
        "target": {
            "type":
            "tensorflow_datasets.core.features.class_label_feature.ClassLabel",
            "num_classes": 10
        }
    }
}

Returns
A dict(type=, content=). Will be forwarded to from_json when reconstructing the feature.

to_json_content

View source

to_json_content() -> tfds.typing.Json

FeatureConnector factory (to overwrite).

This function should be overwritten by the subclass to allow re-importing the feature connector from the config. See existing FeatureConnector for example of implementation.

Returns
Dict containing the FeatureConnector metadata. Will be forwarded to from_json_content when reconstructing the feature.

__contains__

View source

__contains__(
    key: str
) -> bool

__getitem__

View source

__getitem__(
    key
)

Convenience method to access the underlying features.

Class Variables
ALIASES	[]