<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/download/download_manager.py#L146-L776) |

Manages the download and extraction of files, as well as caching.

    tfds.download.DownloadManager(
        *,
        download_dir: epath.PathLike,
        extract_dir: (epath.PathLike | None) = None,
        manual_dir: (epath.PathLike | None) = None,
        manual_dir_instructions: (str | None) = None,
        url_infos: (dict[str, checksums.UrlInfo] | None) = None,
        dataset_name: (str | None) = None,
        force_download: bool = False,
        force_extraction: bool = False,
        force_checksums_validation: bool = False,
        register_checksums: bool = False,
        register_checksums_path: (epath.PathLike | None) = None,
        verify_ssl: bool = True,
        max_simultaneous_downloads: (int | None) = None
    )

Downloaded files are cached under `download_dir`. The file name of downloaded
files follows pattern "{sanitized_url}{content_checksum}.{ext}". Eg:
'cs.toronto.edu_kriz_cifar-100-pythonJDF\[...\]I.tar.gz'.

While a file is being downloaded, it is placed into a directory following a
similar but different pattern:
"{sanitized_url}{url_checksum}.tmp.{uuid}".

When a file is downloaded, a "{fname}.INFO.json" file is created next to it.
This INFO file contains the following information:
{"dataset_names": \["name1", "name2"\],
"urls": \["http://url.of/downloaded_file"\]}

Extracted files/dirs are stored under `extract_dir`. The file name or
directory name is the same as the original name, prefixed with the extraction
method. E.g.
"{extract_dir}/TAR_GZ.cs.toronto.edu_kriz_cifar-100-pythonJDF\[...\]I.tar.gz".

The function members accept either plain value, or values wrapped into list
or dict. Giving a data structure will parallelize the downloads.

#### Example of usage:

    # Sequential download: str -> str
    train_dir = dl_manager.download_and_extract('https://abc.org/train.tar.gz')
    test_dir = dl_manager.download_and_extract('https://abc.org/test.tar.gz')

    # Parallel download: list -> list
    image_files = dl_manager.download(
        ['https://a.org/1.jpg', 'https://a.org/2.jpg', ...])

    # Parallel download: dict -> dict
    data_dirs = dl_manager.download_and_extract({
       'train': 'https://abc.org/train.zip',
       'test': 'https://abc.org/test.zip',
    })
    data_dirs['train']
    data_dirs['test']

For more customization on the download/extraction (ex: passwords, output_name,
...), you can pass a [`tfds.download.Resource`](https://www.tensorflow.org/datasets/api_docs/python/tfds/download/Resource) as argument.

<br />

<br />

<br />

| ## Args ||
|---|---|
| `download_dir` | Path to directory where downloads are stored. |
| `extract_dir` | Path to directory where artifacts are extracted. |
| `manual_dir` | Path to manually downloaded/extracted data directory. |
| `manual_dir_instructions` | Human readable instructions on how to prepare contents of the manual_dir for this dataset. |
| `url_infos` | Urls info for the checksums. |
| `dataset_name` | Name of dataset this instance will be used for. If provided, downloads will contain which datasets they were used for. |
| `force_download` | If True, always \[re\]download. |
| `force_extraction` | If True, always \[re\]extract. |
| `force_checksums_validation` | If True, raises an error if an URL do not have checksums. |
| `register_checksums` | If True, dl checksums aren't checked, but stored into file. |
| `register_checksums_path` | Path were to save checksums. Should be set if register_checksums is True. |
| `verify_ssl` | `bool`, defaults to True. If True, will verify certificate when downloading dataset. |
| `max_simultaneous_downloads` | `int`, optional max number of simultaneous downloads. |

<br />

<br />

<br />

<br />

| ## Raises ||
|---|---|
| `FileNotFoundError` | Raised if the register_checksums_path does not exist. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `download_dir` | <br /> <br /> |
| `downloaded_size` | Returns the total size of downloaded files. |
| `manual_dir` | Returns the directory containing the manually extracted data. |
| `register_checksums` | Returns whether checksums are being computed and recorded to file. |

<br />

## Methods

### `download`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/download/download_manager.py#L654-L667)

    download(
        url_or_urls
    )

Download given url(s).

<br />

<br />

<br />

| Args ||
|---|---|
| `url_or_urls` | url or `list`/`dict` of urls to download and extract. Each url can be a `str` or [`tfds.download.Resource`](https://www.tensorflow.org/datasets/api_docs/python/tfds/download/Resource). |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| `downloaded_path` | `s` <br /> `str`, The downloaded paths matching the given input url_or_urls. |

<br />

### `download_and_extract`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/download/download_manager.py#L733-L754)

    download_and_extract(
        url_or_urls
    )

Download and extract given url_or_urls.

Is roughly equivalent to:

    extracted_paths = dl_manager.extract(dl_manager.download(url_or_urls))

<br />

<br />

<br />

| Args ||
|---|---|
| `url_or_urls` | url or `list`/`dict` of urls to download and extract. Each url can be a `str` or [`tfds.download.Resource`](https://www.tensorflow.org/datasets/api_docs/python/tfds/download/Resource). If not explicitly specified in `Resource`, the extraction method will automatically be deduced from downloaded file name. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| `extracted_path` | `s` <br /> `str`, extracted paths of given URL(s). |

<br />

### `download_checksums`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/download/download_manager.py#L620-L623)

    download_checksums(
        checksums_url
    )

Downloads checksum file from the given URL and adds it to registry.

### `download_kaggle_data`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/download/download_manager.py#L625-L640)

    download_kaggle_data(
        competition_or_dataset: str
    ) -> epath.Path

Download data for a given Kaggle Dataset or competition.

> [!NOTE]
> **Note:** This function requires the Kaggle CLI tool. Read the installation guide at <https://www.kaggle.com/docs/api>

<br />

<br />

<br />

| Args ||
|---|---|
| `competition_or_dataset` | Dataset name (`zillow/zecon`) or competition name (`titanic`) |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| The path to the downloaded files. ||

<br />

### `extract`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/download/download_manager.py#L702-L717)

    extract(
        path_or_paths
    )

Extract given path(s).

<br />

<br />

<br />

| Args ||
|---|---|
| `path_or_paths` | path or `list`/`dict` of path of file to extract. Each path can be a `str` or [`tfds.download.Resource`](https://www.tensorflow.org/datasets/api_docs/python/tfds/download/Resource). If not explicitly specified in `Resource`, the extraction method is deduced from downloaded file name. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| `extracted_path` | `s` <br /> `str`, The extracted paths matching the given input path_or_paths. |

<br />

### `iter_archive`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/download/download_manager.py#L669-L686)

    iter_archive(
        resource: ExtractPath
    ) -> Iterator[tuple[str, typing.BinaryIO]]

Returns iterator over files within archive.

**Important Note**: caller should read files as they are yielded.
Reading out of order is slow.

<br />

<br />

<br />

| Args ||
|---|---|
| `resource` | path to archive or [`tfds.download.Resource`](https://www.tensorflow.org/datasets/api_docs/python/tfds/download/Resource). |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| Generator yielding tuple (path_within_archive, file_obj). ||

<br />