|---| | [ View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.21.0/tensorflow/python/distribute/cross_device_ops.py#L996-L1036) | Hierarchical copy all-reduce implementation of CrossDeviceOps. Inherits From: [`CrossDeviceOps`](https://www.tensorflow.org/api_docs/python/tf/distribute/CrossDeviceOps) #### View aliases **Compat aliases for migration** See [Migration guide](https://www.tensorflow.org/guide/migrate) for more details. [`tf.compat.v1.distribute.HierarchicalCopyAllReduce`](https://www.tensorflow.org/api_docs/python/tf/distribute/HierarchicalCopyAllReduce)
tf.distribute.HierarchicalCopyAllReduce( num_packs=1 ) ### Used in the notebooks | Used in the guide | |---| | - [Distributed training with TensorFlow](https://www.tensorflow.org/guide/distributed_training) | It reduces to one GPU along edges in some hierarchy and broadcasts back to each GPU along the same path. For the batch API, tensors will be repacked or aggregated for more efficient cross-device transportation. This is a reduction created for Nvidia DGX-1 which assumes GPUs connects like that on DGX-1 machine. If you have different GPU inter-connections, it is likely that it would be slower than [`tf.distribute.ReductionToOneDevice`](https://www.tensorflow.org/api_docs/python/tf/distribute/ReductionToOneDevice). For reduces that are not all-reduce, it falls back to [`tf.distribute.ReductionToOneDevice`](https://www.tensorflow.org/api_docs/python/tf/distribute/ReductionToOneDevice). Here is how you can use `HierarchicalCopyAllReduce` in [`tf.distribute.MirroredStrategy`](https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy): strategy = tf.distribute.MirroredStrategy( cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())


| ## Args || |---|---| | `num_packs` | a non-negative integer. The number of packs to split values into. If zero, no packing will be done. |



| ## Raises || |---|---| | ValueError if `num_packs` is negative. ||
## Methods ### `batch_reduce` [View source](https://github.com/tensorflow/tensorflow/blob/v2.21.0/tensorflow/python/distribute/cross_device_ops.py#L400-L447) batch_reduce( reduce_op, value_destination_pairs, options=None ) Reduce values to destinations in batches. See [`tf.distribute.StrategyExtended.batch_reduce_to`](https://www.tensorflow.org/api_docs/python/tf/distribute/StrategyExtended#batch_reduce_to). This can only be called in the cross-replica context.


| Args || |---|---| | `reduce_op` | a [`tf.distribute.ReduceOp`](https://www.tensorflow.org/api_docs/python/tf/distribute/ReduceOp) specifying how values should be combined. | | `value_destination_pairs` | a sequence of (value, destinations) pairs. See [`tf.distribute.CrossDeviceOps.reduce`](https://www.tensorflow.org/api_docs/python/tf/distribute/CrossDeviceOps#reduce) for descriptions. | | `options` | a [`tf.distribute.experimental.CommunicationOptions`](https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/CommunicationOptions). See [`tf.distribute.experimental.CommunicationOptions`](https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/CommunicationOptions) for details. |



| Returns || |---|---| | A list of [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) or [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues), one per pair in `value_destination_pairs`. ||



| Raises || |---|---| | `ValueError` | if `value_destination_pairs` is not an iterable of tuples of [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues) and destinations. |
### `broadcast` [View source](https://github.com/tensorflow/tensorflow/blob/v2.21.0/tensorflow/python/distribute/cross_device_ops.py#L449-L466) broadcast( tensor, destinations ) Broadcast `tensor` to `destinations`. This can only be called in the cross-replica context.


| Args || |---|---| | `tensor` | a [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) like object. The value to broadcast. | | `destinations` | a [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues), a [`tf.Variable`](https://www.tensorflow.org/api_docs/python/tf/Variable), a [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) alike object, or a device string. It specifies the devices to broadcast to. Note that if it's a [`tf.Variable`](https://www.tensorflow.org/api_docs/python/tf/Variable), the value is broadcasted to the devices of that variable, this method doesn't update the variable. |



| Returns || |---|---| | A [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) or [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues). ||
### `reduce` [View source](https://github.com/tensorflow/tensorflow/blob/v2.21.0/tensorflow/python/distribute/cross_device_ops.py#L275-L321) reduce( reduce_op, per_replica_value, destinations, options=None ) Reduce `per_replica_value` to `destinations`. See [`tf.distribute.StrategyExtended.reduce_to`](https://www.tensorflow.org/api_docs/python/tf/distribute/StrategyExtended#reduce_to). This can only be called in the cross-replica context.


| Args || |---|---| | `reduce_op` | a [`tf.distribute.ReduceOp`](https://www.tensorflow.org/api_docs/python/tf/distribute/ReduceOp) specifying how values should be combined. | | `per_replica_value` | a [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues), or a [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) like object. | | `destinations` | a [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues), a [`tf.Variable`](https://www.tensorflow.org/api_docs/python/tf/Variable), a [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) alike object, or a device string. It specifies the devices to reduce to. To perform an all-reduce, pass the same to `value` and `destinations`. Note that if it's a [`tf.Variable`](https://www.tensorflow.org/api_docs/python/tf/Variable), the value is reduced to the devices of that variable, and this method doesn't update the variable. | | `options` | a [`tf.distribute.experimental.CommunicationOptions`](https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/CommunicationOptions). See [`tf.distribute.experimental.CommunicationOptions`](https://www.tensorflow.org/api_docs/python/tf/distribute/experimental/CommunicationOptions) for details. |



| Returns || |---|---| | A [`tf.Tensor`](https://www.tensorflow.org/api_docs/python/tf/Tensor) or [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues). ||



| Raises || |---|---| | `ValueError` | if per_replica_value can't be converted to a [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues) or if destinations is not a string, [`tf.Variable`](https://www.tensorflow.org/api_docs/python/tf/Variable) or [`tf.distribute.DistributedValues`](https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedValues). |