# tf.keras.mixed_precision.experimental.LossScaleOptimizer

<br />

|---|---|
| [![](https://www.tensorflow.org/images/tf_logo_32px.png) TensorFlow 1 version](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer) | [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L49-L333) |

An optimizer that applies loss scaling.

Inherits From: [`Optimizer`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/optimizers/Optimizer)

#### View aliases


**Compat aliases for migration**

See
[Migration guide](https://www.tensorflow.org/guide/migrate) for
more details.

[`tf.compat.v1.keras.mixed_precision.experimental.LossScaleOptimizer`](https://www.tensorflow.org/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer)

<br />

    tf.keras.mixed_precision.experimental.LossScaleOptimizer(
        optimizer, loss_scale
    )

Loss scaling is a process that multiplies the loss by a multiplier called the
loss scale, and divides each gradient by the same multiplier. The pseudocode
for this process is:

    loss = ...
    loss *= loss_scale
    grads = gradients(loss, vars)
    grads /= loss_scale

Mathematically, loss scaling has no effect, but can help avoid numerical
underflow in intermediate gradients when float16 tensors are used. By
multiplying the loss, each intermediate gradient will have the same multiplier
applied.

The loss scale can either be a fixed constant, chosen by the user, or be
dynamically determined. Dynamically determining the loss scale is convenient
as a loss scale does not have to be explicitly chosen. However it reduces
performance.

This optimizer wraps another optimizer and applies loss scaling to it via a
`LossScale`. Loss scaling is applied whenever gradients are
computed, either through `minimize()` or `get_gradients()`. The loss scale is
updated via [`LossScale.update()`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/mixed_precision/experimental/LossScale#update) whenever gradients are applied, either
through `minimize()` or `apply_gradients()`. For example:

    opt = tf.keras.optimizers.SGD(0.1)
    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, "dynamic")
    # 'minimize' applies loss scaling to the loss and updates the loss sale.
    opt.minimize(loss_fn)

If a [`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape) is used to compute gradients instead of
[`LossScaleOptimizer.minimize`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/optimizers/Optimizer#minimize) or [`LossScaleOptimizer.get_gradients`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer#get_gradients), the loss
and gradients must be scaled manually. This can be done by calling
[`LossScaleOptimizer.get_scaled_loss`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer#get_scaled_loss) before passing the loss to
[`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape), and [`LossScaleOptimizer.get_unscaled_gradients`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer#get_unscaled_gradients) after
computing the gradients with [`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape). For example:

    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(...)
    vars = ...
    with tf.GradientTape() as tape:
      loss = ...
      scaled_loss = opt.get_scaled_loss(loss)
    scaled_grads = tape.gradient(scaled_loss, vars)
    grads = opt.get_unscaled_gradients(scaled_grads)
    opt.apply_gradients(zip(grads, vars))  # Loss scale will be updated here

<br />

<br />

<br />

| ## Args ||
|---|---|
| `optimizer` | The Optimizer instance to wrap. |
| `loss_scale` | The loss scale to scale the loss and gradients. This can either be an int/float to use a fixed loss scale, the string "dynamic" to use dynamic loss scaling, or an instance of a LossScale. The string "dynamic" equivalent to passing `DynamicLossScale()`, and passing an int/float is equivalent to passing a FixedLossScale with the given loss scale. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `iterations` | Variable. The number of training steps this Optimizer has run. |
| `learning_rate` | <br /> <br /> |
| `loss_scale` | The `LossScale` instance associated with this optimizer. |
| `lr` | <br /> <br /> |
| `weights` | Returns variables of this Optimizer based on the order created. |

<br />

## Methods

### `add_slot`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L329-L333)

    add_slot(
        var, slot_name, initializer='zeros'
    )

Add a new slot variable for `var`.

### `add_weight`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/optimizer_v2/optimizer_v2.py#L780-L820)

    add_weight(
        name, shape, dtype=None, initializer='zeros', trainable=None,
        synchronization=tf.VariableSynchronization.AUTO,
        aggregation=tf.compat.v1.VariableAggregation.NONE
    )

### `apply_gradients`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L219-L224)

    apply_gradients(
        grads_and_vars, name=None
    )

Apply gradients to variables.

This is the second part of `minimize()`. It returns an `Operation` that
applies gradients.

<br />

<br />

<br />

| Args ||
|---|---|
| `grads_and_vars` | List of (gradient, variable) pairs. |
| `name` | Optional name for the returned operation. Default to the name passed to the `Optimizer` constructor. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| An `Operation` that applies the specified gradients. The `iterations` will be automatically increased by 1. ||

<br />

<br />

<br />

<br />

| Raises ||
|---|---|
| `TypeError` | If `grads_and_vars` is malformed. |
| `ValueError` | If none of the variables have gradients. |

<br />

### `from_config`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L261-L268)

    @classmethod
    from_config(
        config, custom_objects=None
    )

Creates an optimizer from its config.

This method is the reverse of `get_config`,
capable of instantiating the same optimizer from the config
dictionary.

<br />

<br />

<br />

| Arguments ||
|---|---|
| `config` | A Python dictionary, typically the output of get_config. |
| `custom_objects` | A Python dictionary mapping names to additional Python objects used to create this optimizer, such as a function used for a hyperparameter. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| An optimizer instance. ||

<br />

### `get_config`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L253-L259)

    get_config()

Returns the config of the optimimizer.

An optimizer config is a Python dictionary (serializable)
containing the configuration of an optimizer.
The same optimizer can be reinstantiated later
(without any saved state) from this configuration.

<br />

<br />

<br />

| Returns ||
|---|---|
| Python dictionary. ||

<br />

### `get_gradients`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L214-L217)

    get_gradients(
        loss, params
    )

Returns gradients of `loss` with respect to `params`.

<br />

<br />

<br />

| Arguments ||
|---|---|
| `loss` | Loss tensor. |
| `params` | List of variables. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| List of gradient tensors. ||

<br />

<br />

<br />

<br />

| Raises ||
|---|---|
| `ValueError` | In case any gradient cannot be computed (e.g. if gradient function not implemented). |

<br />

### `get_scaled_loss`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L150-L177)

    get_scaled_loss(
        loss
    )

Scales the loss by the loss scale.

This method is only needed if you compute gradients manually, e.g. with
[`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape). In that case, call this method to scale the loss before
passing the loss to [`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape). If you use
[`LossScaleOptimizer.minimize`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/optimizers/Optimizer#minimize) or [`LossScaleOptimizer.get_gradients`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer#get_gradients), loss
scaling is automatically applied and this method is unneeded.

If this method is called, `get_unscaled_gradients` should also be called.
See the [`tf.keras.mixed_precision.experimental.LossScaleOptimizer`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer) doc for
an example.

<br />

<br />

<br />

| Args ||
|---|---|
| `loss` | The loss, which will be multiplied by the loss scale. Can either be a tensor or a callable returning a tensor. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| `loss` multiplied by [`LossScaleOptimizer.loss_scale()`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer#loss_scale). ||

<br />

### `get_slot`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L318-L327)

    get_slot(
        var, slot_name
    )

### `get_slot_names`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L281-L282)

    get_slot_names()

A list of names for this optimizer's slots.

### `get_unscaled_gradients`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L179-L203)

    get_unscaled_gradients(
        grads
    )

Unscales the gradients by the loss scale.

This method is only needed if you compute gradients manually, e.g. with
[`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape). In that case, call this method to unscale the gradients
after computing them with [`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape). If you use
[`LossScaleOptimizer.minimize`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/optimizers/Optimizer#minimize) or [`LossScaleOptimizer.get_gradients`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer#get_gradients), loss
scaling is automatically applied and this method is unneeded.

If this method is called, `get_scaled_loss` should also be called. See
the [`tf.keras.mixed_precision.experimental.LossScaleOptimizer`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer) doc for an
example.

<br />

<br />

<br />

| Args ||
|---|---|
| `grads` | A list of tensors, each which will be divided by the loss scale. Can have None values, which are ignored. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A new list the same size as `grads`, where every non-None value in `grads` is divided by [`LossScaleOptimizer.loss_scale()`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/keras/mixed_precision/experimental/LossScaleOptimizer#loss_scale). ||

<br />

### `get_updates`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/optimizer_v2/optimizer_v2.py#L502-L509)

    get_updates(
        loss, params
    )

### `get_weights`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L291-L292)

    get_weights()

### `minimize`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/optimizer_v2/optimizer_v2.py#L289-L318)

    minimize(
        loss, var_list, grad_loss=None, name=None
    )

Minimize `loss` by updating `var_list`.

This method simply computes gradient using [`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape) and calls
`apply_gradients()`. If you want to process the gradient before applying
then call [`tf.GradientTape`](https://www.tensorflow.org/versions/r2.1/api_docs/python/tf/GradientTape) and `apply_gradients()` explicitly instead
of using this function.

<br />

<br />

<br />

| Args ||
|---|---|
| `loss` | A callable taking no arguments which returns the value to minimize. |
| `var_list` | list or tuple of `Variable` objects to update to minimize `loss`, or a callable returning the list or tuple of `Variable` objects. Use callable when the variable list would otherwise be incomplete before `minimize` since the variables are created at the first time `loss` is called. |
| `grad_loss` | Optional. A `Tensor` holding the gradient computed for `loss`. |
| `name` | Optional name for the returned operation. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| An `Operation` that updates the variables in `var_list`. The `iterations` will be automatically increased by 1. ||

<br />

<br />

<br />

<br />

| Raises ||
|---|---|
| `ValueError` | If some of the variables are not `Variable` objects. |

<br />

### `set_weights`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L294-L295)

    set_weights(
        weights
    )

### `variables`

[View source](https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/mixed_precision/experimental/loss_scale_optimizer.py#L284-L285)

    variables()

Returns variables of this Optimizer based on the order created.