tf.compat.v1.train.experimental.enable_mixed_precision_graph_rewrite
bookmark_borderbookmark
Stay organized with collections
Save and categorize content based on your preferences.
Enable mixed precision via a graph rewrite.
tf.compat.v1.train.experimental.enable_mixed_precision_graph_rewrite(
opt, loss_scale='dynamic'
)
Mixed precision is the use of both float16 and float32 when training a model,
and is used to make the model run faster. This function will use mixed
precision to speed up the execution time of your model when run on a GPU. It
does this by changing the dtype of certain operations in the graph from
float32 to float16.
This function additionally wraps an Optimizer with a LossScaleOptimizer, which
is required to prevent underflow in the float16 tensors during the backwards
pass. An optimizer must be passed to this function, which will then be wrapped
to use loss scaling.
When this function is used, gradients should only be computed and applied with
the returned optimizer, either by calling opt.minimize() or
opt.compute_gradients() followed by opt.apply_gradients(). Gradients
should not be computed with tf.gradients or tf.GradientTape. This is
because the returned optimizer will apply loss scaling, and
tf.gradients/tf.GradientTape will not. If you do directly use
tf.gradients or tf.GradientTape, your model may train to a worse quality.
Currently, mixed precision is only enabled on Volta GPUs and above. TPU
support is coming soon. CPUs are not supported, as CPUs do not run float16
operations faster than float32 operations.
Returns |
|---|
A version of opt that will use loss scaling to prevent underflow.
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2020-10-01 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2020-10-01 UTC."],[],[],null,["# tf.compat.v1.train.experimental.enable_mixed_precision_graph_rewrite\n\n\u003cbr /\u003e\n\n|----------------------------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.0.0/tensorflow/python/training/experimental/mixed_precision.py#L122-L162) |\n\nEnable mixed precision via a graph rewrite. \n\n tf.compat.v1.train.experimental.enable_mixed_precision_graph_rewrite(\n opt, loss_scale='dynamic'\n )\n\nMixed precision is the use of both float16 and float32 when training a model,\nand is used to make the model run faster. This function will use mixed\nprecision to speed up the execution time of your model when run on a GPU. It\ndoes this by changing the dtype of certain operations in the graph from\nfloat32 to float16.\n\nThis function additionally wraps an Optimizer with a LossScaleOptimizer, which\nis required to prevent underflow in the float16 tensors during the backwards\npass. An optimizer must be passed to this function, which will then be wrapped\nto use loss scaling.\n\nWhen this function is used, gradients should only be computed and applied with\nthe returned optimizer, either by calling `opt.minimize()` or\n`opt.compute_gradients()` followed by `opt.apply_gradients()`. Gradients\nshould not be computed with [`tf.gradients`](../../../../../tf/gradients) or [`tf.GradientTape`](../../../../../tf/GradientTape). This is\nbecause the returned optimizer will apply loss scaling, and\n[`tf.gradients`](../../../../../tf/gradients)/[`tf.GradientTape`](../../../../../tf/GradientTape) will not. If you do directly use\n[`tf.gradients`](../../../../../tf/gradients) or [`tf.GradientTape`](../../../../../tf/GradientTape), your model may train to a worse quality.\n\nCurrently, mixed precision is only enabled on Volta GPUs and above. TPU\nsupport is coming soon. CPUs are not supported, as CPUs do not run float16\noperations faster than float32 operations.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `opt` | An instance of a [`tf.keras.optimizers.Optimizer`](../../../../../tf/keras/optimizers/Optimizer) or a `tf.train.Optimizer`. |\n| `loss_scale` | Either an int/float, the string \"dynamic\", or an instance of a [`tf.train.experimental.LossScale`](../../../../../tf/train/experimental/LossScale). The loss scale to use. It is recommended to keep this as its default value of \"dynamic\". |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ------- ||\n|---|---|\n| A version of `opt` that will use loss scaling to prevent underflow. ||\n\n\u003cbr /\u003e"]]
View source on GitHub