tf.contrib.kfac.fisher_factors.EmbeddingInputKroneckerFactor

Class EmbeddingInputKroneckerFactor

Inherits From: DiagonalFactor

Defined in tensorflow/contrib/kfac/python/ops/fisher_factors.py.

FisherFactor for input to an embedding layer.

Given input_ids = [batch_size, input_size] representing indices into an [vocab_size, embedding_size] embedding matrix, approximate input covariance by a diagonal matrix,

Cov(input_ids, input_ids) = (1/batchsize) sum{i} diag(n_hot(input[i]) ** 2).

where n_hot() constructs an n-hot binary vector and diag() constructs a diagonal matrix of size [vocab_size, vocab_size].

Properties

name

Methods

__init__

__init__(
    input_ids,
    vocab_size,
    dtype=None
)

Instantiate EmbeddingInputKroneckerFactor.

Args:

  • input_ids: List of Tensors of shape [batch_size, input_size] and dtype int32. Indices into embedding matrix. List index is tower.
  • vocab_size: int or 0-D Tensor. Maximum value for entries in 'input_ids'.
  • dtype: dtype for covariance statistics. Must be a floating point type. Defaults to float32.

get_cov

get_cov()

get_cov_var

get_cov_var()

Get variable backing this FisherFactor.

May or may not be the same as self.get_cov()

Returns:

Variable of shape self._cov_shape.

instantiate_cov_variables

instantiate_cov_variables()

Makes the internal cov variable(s).

instantiate_inv_variables

instantiate_inv_variables()

left_multiply_matpower

left_multiply_matpower(
    x,
    exp,
    damping_func
)

make_covariance_update_op

make_covariance_update_op(ema_decay)

Constructs and returns the covariance update Op.

Args:

  • ema_decay: The exponential moving average decay (float or Tensor).

Returns:

An Op for updating the covariance Variable referenced by _cov.

make_inverse_update_ops

make_inverse_update_ops()

register_matpower

register_matpower(
    exp,
    damping_func
)

right_multiply_matpower

right_multiply_matpower(
    x,
    exp,
    damping_func
)