tf.contrib.kfac.fisher_factors.NaiveDiagonalFactor

Class NaiveDiagonalFactor

Inherits From: DiagonalFactor

Defined in tensorflow/contrib/kfac/python/ops/fisher_factors.py.

FisherFactor for a diagonal approximation of any type of param's Fisher.

Note that this uses the naive "square the sum estimator", and so is applicable to any type of parameter in principle, but has very high variance.

Methods

__init__

__init__(
    params_grads,
    batch_size
)

Initializes NaiveDiagonalFactor instance.

Args:

  • params_grads: Sequence of Tensors, each with same shape as parameters this FisherFactor corresponds to. For example, the gradient of the loss with respect to parameters.
  • batch_size: int or 0-D Tensor. Size

get_cov

get_cov()

get_cov_var

get_cov_var()

Get variable backing this FisherFactor.

May or may not be the same as self.get_cov()

Returns:

Variable of shape self._cov_shape.

instantiate_covariance

instantiate_covariance()

Instantiates the covariance Variable as the instance member _cov.

left_multiply

left_multiply(
    x,
    damping
)

left_multiply_inverse

left_multiply_inverse(
    x,
    damping
)

make_covariance_update_op

make_covariance_update_op(ema_decay)

Constructs and returns the covariance update Op.

Args:

  • ema_decay: The exponential moving average decay (float or Tensor).

Returns:

An Op for updating the covariance Variable referenced by _cov.

make_inverse_update_ops

make_inverse_update_ops()

register_damped_inverse

register_damped_inverse(damping)

right_multiply

right_multiply(
    x,
    damping
)

right_multiply_inverse

right_multiply_inverse(
    x,
    damping
)