View source on GitHub |
Applies natural exponential decay to the initial learning rate.
tf.compat.v1.train.natural_exp_decay(
learning_rate,
global_step,
decay_steps,
decay_rate,
staircase=False,
name=None
)
When training a model, it is often recommended to lower the learning rate as
the training progresses. This function applies an exponential decay function
to a provided initial learning rate. It requires an global_step
value to
compute the decayed learning rate. You can just pass a TensorFlow variable
that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
decayed_learning_rate = learning_rate * exp(-decay_rate * global_step /
decay_step)
or, if staircase
is True
, as:
decayed_learning_rate = learning_rate * exp(-decay_rate * floor(global_step /
decay_step))
Example: decay exponentially with a base of 0.96:
...
global_step = tf.Variable(0, trainable=False)
learning_rate = 0.1
decay_steps = 5
k = 0.5
learning_rate = tf.compat.v1.train.natural_exp_decay(learning_rate,
global_step,
decay_steps, k)
# Passing global_step to minimize() will increment it at each step.
learning_step = (
tf.compat.v1.train.GradientDescentOptimizer(learning_rate)
.minimize(...my loss..., global_step=global_step)
)
Returns | |
---|---|
A scalar Tensor of the same type as learning_rate . The decayed
learning rate.
|
Raises | |
---|---|
ValueError
|
if global_step is not supplied.
|
eager compatibility
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.