Join us at TensorFlow World, Oct 28-31. Use code TF20 for 20% off select passes.

# tf.keras.mixed_precision.experimental.LossScaleOptimizer

## Class `LossScaleOptimizer`

An optimizer that applies loss scaling.

Inherits From: `Optimizer`

### Aliases:

• Class `tf.compat.v1.keras.mixed_precision.experimental.LossScaleOptimizer`
• Class `tf.compat.v2.keras.mixed_precision.experimental.LossScaleOptimizer`

Loss scaling is a process that multiplies the loss by a multiplier called the loss scale, and divides each gradient by the same multiplier. The pseudocode for this process is:

``````loss = ...
loss *= loss_scale
``````

Mathematically, loss scaling has no effect, but can help avoid numerical underflow in intermediate gradients when float16 tensors are used. By multiplying the loss, each intermediate gradient will have the same multiplier applied.

The loss scale can either be a fixed constant, chosen by the user, or be dynamically determined. Dynamically determining the loss scale is convenient as a loss scale does not have to be explicitly chosen. However it reduces performance.

This optimizer wraps another optimizer and applies loss scaling to it via a `LossScale`. Loss scaling is applied whenever gradients are computed, either through `minimize()` or `get_gradients()`.

## `__init__`

View source

``````__init__(
opt,
loss_scale
)
``````

Initializes this loss scale optimizer.

#### Args:

• `opt`: The Optimizer instance to wrap.
• `loss_scale`: The loss scale to scale the loss and gradients. This can either be an int/float to use a fixed loss scale, the string "dynamic" to use dynamic loss scaling, or an instance of a LossScale. The string "dynamic" equivalent to passing `DynamicLossScale()`, and passing an int/float is equivalent to passing a FixedLossScale with the given loss scale.

## Properties

### `iterations`

Variable. The number of training steps this Optimizer has run.

### `weights`

Returns variables of this Optimizer based on the order created.

## Methods

### `add_slot`

View source

``````add_slot(
var,
slot_name,
initializer='zeros'
)
``````

Add a new slot variable for `var`.

### `add_weight`

View source

``````add_weight(
name,
shape,
dtype=None,
initializer='zeros',
trainable=None,
synchronization=tf.VariableSynchronization.AUTO,
aggregation=tf.VariableAggregation.NONE
)
``````

### `apply_gradients`

View source

``````apply_gradients(
name=None
)
``````

### `from_config`

View source

``````@classmethod
from_config(
cls,
config,
custom_objects=None
)
``````

### `get_config`

View source

``````get_config()
``````

### `get_gradients`

View source

``````get_gradients(
loss,
params
)
``````

### `get_slot`

View source

``````get_slot(
var,
slot_name
)
``````

### `get_slot_names`

View source

``````get_slot_names()
``````

A list of names for this optimizer's slots.

### `get_updates`

View source

``````get_updates(
loss,
params
)
``````

### `get_weights`

View source

``````get_weights()
``````

### `minimize`

View source

``````minimize(
loss,
var_list,
name=None
)
``````

Minimize `loss` by updating `var_list`.

This method simply computes gradient using `tf.GradientTape` and calls `apply_gradients()`. If you want to process the gradient before applying then call `tf.GradientTape` and `apply_gradients()` explicitly instead of using this function.

#### Args:

• `loss`: A callable taking no arguments which returns the value to minimize.
• `var_list`: list or tuple of `Variable` objects to update to minimize `loss`, or a callable returning the list or tuple of `Variable` objects. Use callable when the variable list would otherwise be incomplete before `minimize` since the variables are created at the first time `loss` is called.
• `grad_loss`: Optional. A `Tensor` holding the gradient computed for `loss`.
• `name`: Optional name for the returned operation.

#### Returns:

An Operation that updates the variables in `var_list`. If `global_step` was not `None`, that operation also increments `global_step`.

#### Raises:

• `ValueError`: If some of the variables are not `Variable` objects.

### `set_weights`

View source

``````set_weights(weights)
``````

### `variables`

View source

``````variables()
``````

Returns variables of this Optimizer based on the order created.