Missed TensorFlow World? Check out the recap. Learn more

nsl.lib.decay_over_time

View source on GitHub

Returns a decayed value of init_value over time.

nsl.lib.decay_over_time(
    global_step,
    decay_config,
    init_value=1.0
)

When training a model with a regularizer, the objective function can be formulated as the following:

\(objective = \lambda_1 * loss + \lambda_2 * regularization\)

This function can be used for three cases:

  1. Incrementally diminishing the importance of the loss term, by applying a decay function to the
    \(\lambda_1\)
    over time. We'll denote this by writing
    \(\lambda_1\)
    = decay_over_time(init_value).
  2. Incrementally increasing the importance of the regularization term, by setting
    \(\lambda_2\)
    = init_value - decay_over_time(init_value).
  3. Combining the above two cases, namely, setting
    \(\lambda_1\)
    = decay_over_time(init_value) and
    \(\lambda_2\)
    = init_value - decay_over_time(init_value).

This function requires a global_step value to compute the decayed value.

Args:

  • global_step: A scalar int32 or int64 Tensor or a Python number. Must be positive.
  • decay_config: A nsl.configs.DecayConfig for computing the decay value.
  • init_value: A scalar Tensor to set the initial value to be decayed.

Returns:

A scalar float Tensor.