Join us at TensorFlow World, Oct 28-31. Use code TF20 for 20% off select passes. Register now

nsl.lib.decay_over_time

View source on GitHub

Returns a decayed value of init_value over time.

nsl.lib.decay_over_time(
    global_step,
    decay_config,
    init_value=1.0
)

When training a model with a regularizer, the objective function can be formulated as the following:

\(objective = \lambda_1 * loss + \lambda_2 * regularization\)

This function can be used for three cases: (1) Incrementally diminish the importance of loss, by applying a decay function to the \lambda_1 over time. (2) Incrementally increase the importance of regularization, by setting the \lambda_2 = init_value - decay_over_time(init_value). (3) Combine the above two cases. Let \lambda_1 = decay_over_time(init_value) and \lambda_2 = init_value - decay_over_time(init_value).

This function requires a global_step value to compute the decayed value.

Args:

  • global_step: A scalar int32 or int64 Tensor or a Python number. Must be positive.
  • decay_config: DecayConfig contains the following configs (or hyper-parameters) for computing the decay value: (a) decay_type: Type of decay function to apply. (b) decay_steps: A scalar int32 or int64 Tensor or a Python number. Must be positive. The decay will be applied every 'decay_steps'. (c) decay_rate: A scalar float32 or float64 Tensor or a Python number. (d) min_value: minimal acceptable value after applying decay to init_value.
  • init_value: A scalar Tensor to set the init value to be decayed.

Returns:

A scalar float Tensor.