# tf.distribute.experimental.CollectiveHints

<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.13.1/tensorflow/python/distribute/collective_util.py#L168-L233) |

Hints for collective operations like AllReduce.

#### View aliases


**Compat aliases for migration**

See
[Migration guide](https://www.tensorflow.org/guide/migrate) for
more details.

[`tf.compat.v1.distribute.experimental.CollectiveHints`](https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/CollectiveHints)

<br />

    tf.distribute.experimental.CollectiveHints(
        bytes_per_pack=0, timeout_seconds=None
    )

This can be passed to methods like
`tf.distribute.get_replica_context().all_reduce()` to optimize collective
operation performance. Note that these are only hints, which may or may not
change the actual behavior. Some options only apply to certain strategy and
are ignored by others.

One common optimization is to break gradients all-reduce into multiple packs
so that weight updates can overlap with gradient all-reduce.

#### Examples:

- bytes_per_pack

    hints = tf.distribute.experimental.CollectiveHints(
        bytes_per_pack=50 * 1024 * 1024)
    grads = tf.distribute.get_replica_context().all_reduce(
        'sum', grads, experimental_hints=hints)
    optimizer.apply_gradients(zip(grads, vars),
        experimental_aggregate_gradients=False)

- timeout_seconds

    strategy = tf.distribute.MirroredStrategy()
    hints = tf.distribute.experimental.CollectiveHints(
        timeout_seconds=120.0)
    try:
      strategy.reduce("sum", v, axis=None, experimental_hints=hints)
    except tf.errors.DeadlineExceededError:
      do_something()

<br />

<br />

<br />

| ## Args ||
|---|---|
| `bytes_per_pack` | a non-negative integer. Breaks collective operations into packs of certain size. If it's zero, the value is determined automatically. This only applies to all-reduce with `MultiWorkerMirroredStrategy` currently. |
| `timeout_seconds` | a float or None, timeout in seconds. If not None, the collective raises [`tf.errors.DeadlineExceededError`](https://www.tensorflow.org/versions/r2.13/api_docs/python/tf/errors/DeadlineExceededError) if it takes longer than this timeout. This can be useful when debugging hanging issues. This should only be used for debugging since it creates a new thread for each collective, i.e. an overhead of `timeout_seconds * num_collectives_per_second` more threads. This only works for [`tf.distribute.experimental.MultiWorkerMirroredStrategy`](https://www.tensorflow.org/versions/r2.13/api_docs/python/tf/distribute/experimental/MultiWorkerMirroredStrategy). |

<br />

<br />

<br />

<br />

| ## Raises ||
|---|---|
| `ValueError` | When arguments have invalid value. |

<br />