tfds.core.ShardedFileTemplate

View source on GitHub

Template to produce filenames for sharded datasets.

tfds.core.ShardedFileTemplate(
    data_dir: tfds.core.Path,
    template: str = DEFAULT_FILENAME_TEMPLATE,
    dataset_name: Optional[str] = None,
    split: Optional[str] = None,
    filetype_suffix: Optional[str] = None
)

Attributes
data_dir	the directory that contains the files for the shards.
template	template of the sharded files, e.g. '\({SPLIT}/data.\){FILEFORMAT}-${SHARD_INDEX}'.
dataset_name	the name of the dataset.
split	the split of the dataset.
filetype_suffix	the filetype suffix to denote the type of file. For example, tfrecord.

Methods

filepath_prefix

View source

filepath_prefix() -> str

replace

View source

replace(
    **kwargs
) -> 'ShardedFileTemplate'

Returns a copy of the ShardedFileTemplate with updated attributes.

sharded_filenames

View source

sharded_filenames(
    num_shards: int
) -> List[str]

sharded_filepath

View source

sharded_filepath(
    *,
    shard_index: int,
    num_shards: tfds.typing.Dim
) -> tfds.core.Path

Returns the filename (including full path if data_dir is set) for the given shard.

sharded_filepaths

View source

sharded_filepaths(
    num_shards: int
) -> List[tfds.core.Path]

sharded_filepaths_pattern

View source

sharded_filepaths_pattern(
    *,
    num_shards: tfds.typing.Dim = None
) -> str

Returns a pattern describing all the file paths captured by this template.

If num_shards is given, then it returns '/path/dataset_name-split.fileformat@num_shards. Ifnum_shardsis not given, then it returns '/path/dataset_name-split.fileformat*.

Args
num_shards	optional specification of the number of shards.

Returns
the pattern describing all shards captured by this template.

__eq__

__eq__(
    other
)

Class Variables
dataset_name	None
filetype_suffix	None
split	None
template	'{DATASET}-{SPLIT}.{FILEFORMAT}-{SHARD_X_OF_Y}'