AutoShardDataset

public final class AutoShardDataset

Creates a dataset that shards the input dataset.

Creates a dataset that shards the input dataset by num_workers, returning a sharded dataset for the index-th worker. This attempts to automatically shard a dataset by examining the Dataset graph and inserting a shard op before the inputs to a reader Dataset (e.g. CSVDataset, TFRecordDataset).

This dataset will throw a NotFound error if we cannot shard the dataset automatically.

Nested Classes

class AutoShardDataset.Options Optional attributes for AutoShardDataset

Constants

String OP_NAME The name of this op, as known by TensorFlow core engine

Public Methods

Output < TType >
asOutput ()
Returns the symbolic handle of the tensor.
static AutoShardDataset.Options
autoShardPolicy (Long autoShardPolicy)
static AutoShardDataset
create ( Scope scope, Operand <?> inputDataset, Operand < TInt64 > numWorkers, Operand < TInt64 > index, List<Class<? extends TType >> outputTypes, List< Shape > outputShapes, Options... options)
Factory method to create a class wrapping a new AutoShardDataset operation.
Output <?>

Inherited Methods

Constants

public static final String OP_NAME

The name of this op, as known by TensorFlow core engine

Constant Value: "ExperimentalAutoShardDataset"

Public Methods

public Output < TType > asOutput ()

Returns the symbolic handle of the tensor.

Inputs to TensorFlow operations are outputs of another TensorFlow operation. This method is used to obtain a symbolic handle that represents the computation of the input.

public static AutoShardDataset.Options autoShardPolicy (Long autoShardPolicy)

public static AutoShardDataset create ( Scope scope, Operand <?> inputDataset, Operand < TInt64 > numWorkers, Operand < TInt64 > index, List<Class<? extends TType >> outputTypes, List< Shape > outputShapes, Options... options)

Factory method to create a class wrapping a new AutoShardDataset operation.

Parameters
scope current scope
inputDataset A variant tensor representing the input dataset.
numWorkers A scalar representing the number of workers to distribute this dataset across.
index A scalar representing the index of the current worker out of num_workers.
options carries optional attributes values
Returns
  • a new instance of AutoShardDataset

public Output <?> handle ()