ML Community Day is November 9! Join us for updates from TensorFlow, JAX, and more Learn more

tf.keras.mixed_precision.experimental.LossScaleOptimizer

An deprecated optimizer that applies loss scaling.

Inherits From: LossScaleOptimizer, Optimizer

This class is identical to the non-experimental keras.mixed_precision.LossScaleOptimizer except its constructor takes different arguments. For this class (the experimental version), the constructor takes a loss_scale argument. For the non-experimental class, the constructor encodes the loss scaling information in multiple arguments. Note that unlike this class, the non-experimental class does not accept a tf.compat.v1.mixed_precision.LossScale, which is deprecated.

If you currently use this class, you should switch to the non-experimental tf.keras.mixed_precision.LossScaleOptimizer instead. We show several examples of converting the use of the experimental class to the equivalent non-experimental class.

# In all of the the examples below, `opt1` and `opt2` are identical
opt1 = tf.keras.mixed_precision.experimental.LossScaleOptimizer(
    tf.keras.optimizers.SGD(), loss_scale='dynamic')
opt2 = tf.keras.mixed_precision.LossScaleOptimizer(
    tf.keras.optimizers.SGD())
assert opt1.get_config() == opt2.get_config()
opt1 = tf.keras.mixed_precision.experimental.LossScaleOptimizer(
    tf.keras.optimizers.SGD(), loss_scale=123)
# dynamic=False indicates to use fixed loss scaling. initial_scale=123
# refers to the initial loss scale, which is the single fixed loss scale
# when dynamic=False.
opt2 = tf.keras.mixed_precision.LossScaleOptimizer(
    tf.keras.optimizers.SGD(), dynamic=False, initial_scale=123)
assert opt1.get_config() == opt2.get_config()
loss_scale = tf.compat.v1.mixed_precision.experimental.DynamicLossScale(
    initial_loss_scale=2048, increment_period=500)
opt1 = tf.keras.mixed_precision.experimental.LossScaleOptimizer(
    tf.keras.optimizers.SGD(), loss_scale=loss_scale)
opt2 = tf.keras.mixed_precision.LossScaleOptimizer(
    tf.keras.optimizers.SGD(), initial_scale=2048,
    dynamic_growth_steps=500)
assert opt1.get_config() == opt2.get_config()

Make sure to also switch from this class to the non-experimental class in isinstance checks, if you have any. If you do not do this, your model may run into hard-to-debug issues, as the experimental LossScaleOptimizer subclasses the non-experimental LossScaleOptimizer, but not vice versa. It is safe to switch isinstance checks to the non-experimental