tf.contrib.kfac.fisher_factors.FullyConnectedDiagonalFactor

Class FullyConnectedDiagonalFactor

Inherits From: DiagonalFactor

Defined in tensorflow/contrib/kfac/python/ops/fisher_factors.py.

FisherFactor for a diagonal approx of a fully-connected layer's Fisher.

Given in = [batch_size, input_size] and out_grad = [batch_size, output_size], approximates the covariance as,

Cov(in, out) = (1/batchsize) sum{i} outer(in[i], out_grad[i]) ** 2.0

where the square is taken element-wise.

Properties

name

Methods

__init__

__init__(
    inputs,
    outputs_grads,
    has_bias=False
)

Instantiate FullyConnectedDiagonalFactor.

Args:

  • inputs: List of Tensors of shape [batch_size, input_size]. Inputs to this layer. List index is towers.
  • outputs_grads: List of Tensors, each of shape [batch_size, output_size], which are the gradients of the loss with respect to the layer's outputs. First index is source, second is tower.

  • has_bias: bool. If True, append '1' to each input.

get_cov

get_cov()

get_cov_var

get_cov_var()

Get variable backing this FisherFactor.

May or may not be the same as self.get_cov()

Returns:

Variable of shape self._cov_shape.

instantiate_cov_variables

instantiate_cov_variables()

Makes the internal cov variable(s).

instantiate_inv_variables

instantiate_inv_variables()

left_multiply_matpower

left_multiply_matpower(
    x,
    exp,
    damping_func
)

make_covariance_update_op

make_covariance_update_op(ema_decay)

make_inverse_update_ops

make_inverse_update_ops()

register_matpower

register_matpower(
    exp,
    damping_func
)

right_multiply_matpower

right_multiply_matpower(
    x,
    exp,
    damping_func
)