tfds.download.DownloadConfig

View source on GitHub

Class DownloadConfig

Configuration for tfds.core.DatasetBuilder.download_and_prepare.

__init__

View source

__init__(
    extract_dir=None,
    manual_dir=None,
    download_mode=None,
    compute_stats=None,
    max_examples_per_split=None,
    register_checksums=False,
    beam_runner=None,
    beam_options=None,
    try_download_gcs=True
)

Constructs a DownloadConfig.

Args:

  • extract_dir: str, directory where extracted files are stored. Defaults to "/extracted".
  • manual_dir: str, read-only directory where manually downloaded/extracted data is stored. Defaults to "/manual".
  • download_mode: tfds.GenerateMode, how to deal with downloads or data that already exists. Defaults to REUSE_DATASET_IF_EXISTS, which will reuse both downloads and data if it already exists.
  • compute_stats: tfds.download.ComputeStats, whether to compute statistics over the generated data. Defaults to AUTO.
  • max_examples_per_split: int, optional max number of examples to write into each split (used for testing).
  • register_checksums: bool, defaults to False. If True, checksum of downloaded files are recorded.
  • beam_runner: Runner to pass to beam.Pipeline, only used for datasets based on Beam for the generation.
  • beam_options: PipelineOptions to pass to beam.Pipeline, only used for datasets based on Beam for the generation.
  • try_download_gcs: bool, defaults to True. If True, prepared dataset will be downloaded from GCS, when available. If False, dataset will be downloaded and prepared from scratch.