Join us at TensorFlow World, Oct 28-31. Use code TF20 for 20% off select passes. Register now


View source on GitHub

Returns a decayed value of init_value over time.


When training a model with a regularizer, the objective function can be formulated as the following:

\(objective = \lambda_1 * loss + \lambda_2 * regularization\)

This function can be used for three cases: (1) Incrementally diminish the importance of loss, by applying a decay function to the \lambda_1 over time. (2) Incrementally increase the importance of regularization, by setting the \lambda_2 = init_value - decay_over_time(init_value). (3) Combine the above two cases. Let \lambda_1 = decay_over_time(init_value) and \lambda_2 = init_value - decay_over_time(init_value).

This function requires a global_step value to compute the decayed value.


  • global_step: A scalar int32 or int64 Tensor or a Python number. Must be positive.
  • decay_config: DecayConfig contains the following configs (or hyper-parameters) for computing the decay value: (a) decay_type: Type of decay function to apply. (b) decay_steps: A scalar int32 or int64 Tensor or a Python number. Must be positive. The decay will be applied every 'decay_steps'. (c) decay_rate: A scalar float32 or float64 Tensor or a Python number. (d) min_value: minimal acceptable value after applying decay to init_value.
  • init_value: A scalar Tensor to set the init value to be decayed.


A scalar float Tensor.