tf.contrib.all_reduce.build_recursive_hd_all_reduce

tf.contrib.all_reduce.build_recursive_hd_all_reduce(
    input_tensors,
    red_op,
    un_op=None
)

Defined in tensorflow/contrib/all_reduce/python/all_reduce.py.

Construct a subgraph for recursive halving-doubling all-reduce.

The recursive halving-doubling algorithm is described in http://www.mcs.anl.gov/~thakur/papers/ijhpca-coll.pdf

The concept is to arrange the participating n devices in a linear sequence where devices exchange data pairwise with one other device in each round. During the gather phase there are lg(n) rounds where devices exchange increasingly smaller sub-tensors with another device at increasingly greater distances, until at the top each device has 1/n of the fully reduced values. During the scatter phase each device exchanges its fully reduced sub-tensor (which doubles in length at each round) with one other device at increasingly smaller distances until each device has all of the fully reduced values.

Args:

  • input_tensors: list of T tf.Tensor to be elementwise reduced.
  • red_op: a binary elementwise reduction Op.
  • un_op: an optional unary elementwise Op to apply to reduced values.

Returns:

list of T tf.Tensor which are the fully reduced tensors, one at each device of input_tensors.

Raises:

  • ValueError: num_devices not a power of 2, or tensor len not divisible by 2 the proper number of times.