Defined in tensorflow/contrib/all_reduce/python/

Construct a subgraph for recursive halving-doubling all-reduce.

The recursive halving-doubling algorithm is described in

The concept is to arrange the participating n devices in a linear sequence where devices exchange data pairwise with one other device in each round. During the gather phase there are lg(n) rounds where devices exchange increasingly smaller sub-tensors with another device at increasingly greater distances, until at the top each device has 1/n of the fully reduced values. During the scatter phase each device exchanges its fully reduced sub-tensor (which doubles in length at each round) with one other device at increasingly smaller distances until each device has all of the fully reduced values.


  • input_tensors: list of T tf.Tensor to be elementwise reduced.
  • red_op: a binary elementwise reduction Op.
  • un_op: an optional unary elementwise Op to apply to reduced values.


list of T tf.Tensor which are the fully reduced tensors, one at each device of input_tensors.


  • ValueError: num_devices not a power of 2, or tensor len not divisible by 2 the proper number of times.