tfds.folder_dataset.TranslateFolder

View source on GitHub

Generic text translation dataset created from manual directory.

Inherits From: DatasetBuilder

tfds.folder_dataset.TranslateFolder(
    root_dir: str
)

The directory content should be as followed:

path/to/my_data/
  lang1.train.txt
  lang2.train.txt
  lang1.test.txt
  lang2.test.txt
  ...

Each files should have one example per line. Line order should match between files.

To use it:

builder = tfds.TranslateFolder(root_dir='path/to/my_data/')
print(builder.info)  # Splits, num examples,... are automatically calculated
ds = builder.as_dataset(split='train', shuffle_files=True)

Methods

as_data_source

View source

as_data_source(
    split: Optional[Tree[splits_lib.SplitArg]] = None,
    *,
    decoders: Optional[TreeDict[decode.partial_decode.DecoderArg]] = None
) -> ListOrTreeOrElem[Sequence[Any]]

Constructs an ArrayRecordDataSource.

as_dataset

View source

as_dataset(
    split: Optional[Tree[splits_lib.SplitArg]] = None,
    *,
    batch_size: Optional[int] = None,
    shuffle_files: bool = False,
    decoders: Optional[TreeDict[decode.partial_decode.DecoderArg]] = None,
    read_config: Optional[read_config_lib.ReadConfig] = None,
    as_supervised: bool = False
)

Constructs a tf.data.Dataset.

Callers must pass arguments as keyword arguments.

The output types vary depending on the parameters. Examples:

builder = tfds.builder('imdb_reviews')
builder.download_and_prepare()

# Default parameters: Returns the dict of tf.data.Dataset
ds_all_dict = builder.as_dataset()
assert isinstance(ds_all_dict, dict)
print(ds_all_dict.keys())  # ==> ['test', 'train', 'unsupervised']

assert isinstance(ds_all_dict['test'], tf.data.Dataset)
# Each dataset (test, train, unsup.) consists of dictionaries
# {'label': <tf.Tensor: .. dtype=int64, numpy=1>,
#  'text': <tf.Tensor: .. dtype=string, numpy=b"I've watched the movie ..">}
# {'label': <tf.Tensor: .. dtype=int64, numpy=1>,
#  'text': <tf.Tensor: .. dtype=string, numpy=b'If you love Japanese ..'>}

# With as_supervised: tf.data.Dataset only contains (feature, label) tuples
ds_all_supervised = builder.as_dataset(as_supervised=True)
assert isinstance(ds_all_supervised, dict)
print(ds_all_supervised.keys())  # ==> ['test', 'train', 'unsupervised']

assert isinstance(ds_all_supervised['test'], tf.data.Dataset)
# Each dataset (test, train, unsup.) consists of tuples (text, label)
# (<tf.Tensor: ... dtype=string, numpy=b"I've watched the movie ..">,
#  <tf.Tensor: ... dtype=int64, numpy=1>)
# (<tf.Tensor: ... dtype=string, numpy=b"If you love Japanese ..">,
#  <tf.Tensor: ... dtype=int64, numpy=1>)

# Same as above plus requesting a particular split
ds_test_supervised = builder.as_dataset(as_supervised=True, split='test')
assert isinstance(ds_test_supervised, tf.data.Dataset)
# The dataset consists of tuples (text, label)
# (<tf.Tensor: ... dtype=string, numpy=b"I've watched the movie ..">,
#  <tf.Tensor: ... dtype=int64, numpy=1>)
# (<tf.Tensor: ... dtype=string, numpy=b"If you love Japanese ..">,
#  <tf.Tensor: ... dtype=int64, numpy=1>)

dataset_info_from_configs

View source

dataset_info_from_configs(
    **kwargs
)

Returns the DatasetInfo using given kwargs and config files.

Sub-class should call this and add information not present in config files using kwargs directly passed to tfds.core.DatasetInfo object.

If information is present both in passed arguments and config files, config files will prevail.

download_and_prepare

View source

download_and_prepare(
    **kwargs
)

Downloads and prepares dataset for reading.

get_default_builder_config

View source

get_default_builder_config() -> Optional[BuilderConfig]

Returns the default builder config if there is one.

Note that for dataset builders that cannot use the cls.BUILDER_CONFIGS, we need a method that uses the instance to get BUILDER_CONFIGS and DEFAULT_BUILDER_CONFIG_NAME.

get_metadata

View source

@classmethod
get_metadata() -> dataset_metadata.DatasetMetadata

Returns metadata (README, CITATIONS, ...) specified in config files.

The config files are read from the same package where the DatasetBuilder has been defined, so those metadata might be wrong for legacy builders.

get_reference

View source

get_reference(
    namespace: Optional[str] = None
) -> naming.DatasetReference

Returns a reference to the dataset produced by this dataset builder.

Includes the config if specified, the version, and the data_dir that should contain this dataset.

is_prepared

View source

is_prepared() -> bool

Returns whether this dataset is already downloaded and prepared.

}

}