Creates a dataset that shards the input dataset.
tf.raw_ops.AutoShardDataset(
input_dataset,
num_workers,
index,
output_types,
output_shapes,
auto_shard_policy=0,
num_replicas=0,
name=None
)
Creates a dataset that shards the input dataset by num_workers, returning a sharded dataset for the index-th worker. This attempts to automatically shard a dataset by examining the Dataset graph and inserting a shard op before the inputs to a reader Dataset (e.g. CSVDataset, TFRecordDataset).
This dataset will throw a NotFound error if we cannot shard the dataset automatically.
Args | |
---|---|
input_dataset
|
A Tensor of type variant .
A variant tensor representing the input dataset.
|
num_workers
|
A Tensor of type int64 .
A scalar representing the number of workers to distribute this dataset across.
|
index
|
A Tensor of type int64 .
A scalar representing the index of the current worker out of num_workers.
|
output_types
|
A list of tf.DTypes that has length >= 1 .
|
output_shapes
|
A list of shapes (each a tf.TensorShape or list of ints ) that has length >= 1 .
|
auto_shard_policy
|
An optional int . Defaults to 0 .
|
num_replicas
|
An optional int . Defaults to 0 .
|
name
|
A name for the operation (optional). |
Returns | |
---|---|
A Tensor of type variant .
|