|TensorFlow 1 version||View source on GitHub|
Hierarchical copy all-reduce implementation of CrossDeviceOps.
Compat aliases for migration
See Migration guide for more details.
tf.distribute.HierarchicalCopyAllReduce( num_packs=1 )
Used in the notebooks
|Used in the guide|
It reduces to one GPU along edges in some hierarchy and broadcasts back to each GPU along the same path. For the batch API, tensors will be repacked or aggregated for more efficient cross-device transportation.
This is a reduction created for Nvidia DGX-1 which assumes GPUs connects like
that on DGX-1 machine. If you have different GPU inter-connections, it is
likely that it would be slower than
For reduces that are not all-reduce, it fal