tfds.split_for_jax_process
Stay organized with collections
Save and categorize content based on your preferences.
Returns the subsplit of the data for the process.
tfds.split_for_jax_process(
split: str,
*,
process_index: tfds.typing.Dim
= None,
process_count: tfds.typing.Dim
= None,
drop_remainder: bool = False
) -> tfds.typing.SplitArg
In distributed setting, all process/hosts should get a non-overlapping,
equally sized slice of the entire data. This function takes as input a split
and extracts the slice for the current process index.
Usage:
tfds.load(..., split=tfds.split_for_jax_process('train'))
This funtion is an alias for:
tfds.even_splits(split, n=jax.process_count())[jax.process_index()]
By default, if examples can't be evenly distributed across processes, you can
drop extra examples with drop_remainder=True
.
Args |
split
|
Split to distribute across host (e.g. train[75%:] ,
train[:800]+validation[:100] ).
|
process_index
|
Process index in [0, count) . Defaults to
jax.process_index() .
|
process_count
|
Number of processes. Defaults to jax.process_count() .
|
drop_remainder
|
Drop examples if the number of examples in the datasets is
not evenly divisible by n . If False , examples are distributed evenly
across subsplits, starting by the first. For example, if there is 11
examples with n=3 , splits will contain [4, 4, 3] examples
respectivelly.
|
Returns |
subsplit
|
The sub-split of the given split for the current
process_index .
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-04-26 UTC.
[{
"type": "thumb-down",
"id": "missingTheInformationINeed",
"label":"Missing the information I need"
},{
"type": "thumb-down",
"id": "tooComplicatedTooManySteps",
"label":"Too complicated / too many steps"
},{
"type": "thumb-down",
"id": "outOfDate",
"label":"Out of date"
},{
"type": "thumb-down",
"id": "samplesCodeIssue",
"label":"Samples / code issue"
},{
"type": "thumb-down",
"id": "otherDown",
"label":"Other"
}]
[{
"type": "thumb-up",
"id": "easyToUnderstand",
"label":"Easy to understand"
},{
"type": "thumb-up",
"id": "solvedMyProblem",
"label":"Solved my problem"
},{
"type": "thumb-up",
"id": "otherUp",
"label":"Other"
}]
{"lastModified": "Last updated 2024-04-26 UTC."}
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-04-26 UTC."],[],[]]