![]() |
Template to produce filenames for sharded datasets.
tfds.core.ShardedFileTemplate(
data_dir: tfds.core.Path
,
template: str = DEFAULT_FILENAME_TEMPLATE,
dataset_name: Optional[str] = None,
split: Optional[str] = None,
filetype_suffix: Optional[str] = None
)
Attributes | |
---|---|
data_dir
|
the directory that contains the files for the shards. |
template
|
template of the sharded files, e.g. '\({SPLIT}/data.\){FILEFORMAT}-${SHARD_INDEX}'. |
dataset_name
|
the name of the dataset. |
split
|
the split of the dataset. |
filetype_suffix
|
the filetype suffix to denote the type of file. For
example, tfrecord .
|
Methods
filepath_prefix
filepath_prefix() -> str
replace
replace(
**kwargs
) -> 'ShardedFileTemplate'
Returns a copy of the ShardedFileTemplate
with updated attributes.
sharded_filenames
sharded_filenames(
num_shards: int
) -> List[str]
sharded_filepath
sharded_filepath(
*,
shard_index: int,
num_shards: tfds.typing.Dim
) -> tfds.core.Path
Returns the filename (including full path if data_dir
is set) for the given shard.
sharded_filepaths
sharded_filepaths(
num_shards: int
) -> List[tfds.core.Path
]
sharded_filepaths_pattern
sharded_filepaths_pattern(
*,
num_shards: tfds.typing.Dim
= None
) -> str
Returns a pattern describing all the file paths captured by this template.
If num_shards
is given, then it returns
'/path/dataset_name-split.fileformat@num_shards.
If
num_shardsis not given, then it returns
'/path/dataset_name-split.fileformat*
.
Args | |
---|---|
num_shards
|
optional specification of the number of shards. |
Returns | |
---|---|
the pattern describing all shards captured by this template. |
__eq__
__eq__(
other
)
Class Variables | |
---|---|
dataset_name |
None
|
filetype_suffix |
None
|
split |
None
|
template |
'{DATASET}-{SPLIT}.{FILEFORMAT}-{SHARD_X_OF_Y}'
|