tf.contrib.all_reduce.build_recursive_hd_all_reduce( input_tensors, red_op, un_op=None )
Construct a subgraph for recursive halving-doubling all-reduce.
The recursive halving-doubling algorithm is described in http://www.mcs.anl.gov/~thakur/papers/ijhpca-coll.pdf
The concept is to arrange the participating n devices in a linear sequence where devices exchange data pairwise with one other device in each round. During the gather phase there are lg(n) rounds where devices exchange increasingly smaller sub-tensors with another device at increasingly greater distances, until at the top each device has 1/n of the fully reduced values. During the scatter phase each device exchanges its fully reduced sub-tensor (which doubles in length at each round) with one other device at increasingly smaller distances until each device has all of the fully reduced values.
input_tensors: list of T
tf.Tensorto be elementwise reduced.
red_op: a binary elementwise reduction Op.
un_op: an optional unary elementwise Op to apply to reduced values.
list of T
tf.Tensor which are the fully reduced tensors, one
at each device of input_tensors.
ValueError: num_devices not a power of 2, or tensor len not divisible by 2 the proper number of times.