TensorFlow 1 version | View source on GitHub |
Clips values of multiple tensors by the ratio of the sum of their norms.
tf.clip_by_global_norm(
t_list, clip_norm, use_norm=None, name=None
)
Given a tuple or list of tensors t_list
, and a clipping ratio clip_norm
,
this operation returns a list of clipped tensors list_clipped
and the global norm (global_norm
) of all tensors in t_list
. Optionally,
if you've already computed the global norm for t_list
, you can specify
the global norm with use_norm
.
To perform the clipping, the values t_list[i]
are set to:
t_list[i] * clip_norm / max(global_norm, clip_norm)
where:
global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))
If clip_norm > global_norm
then the entries in t_list
remain as they are,
otherwise they're all shrunk by the global ratio.
If global_norm == infinity
then the entries in t_list
are all set to NaN
to signal that an error occurred.
Any of the entries of t_list
that are of type None
are ignored.
This is the correct way to perform gradient clipping (Pascanu et al., 2012).
However, it is slower than clip_by_norm()
because all the parameters must be
ready before the clipping operation can be performed.
Args | |
---|---|
t_list
|
A tuple or list of mixed Tensors , IndexedSlices , or None.
|
clip_norm
|
A 0-D (scalar) Tensor > 0. The clipping ratio.
|
use_norm
|
A 0-D (scalar) Tensor of type float (optional). The global
norm to use. If not provided, global_norm() is used to compute the norm.
|
name
|
A name for the operation (optional). |
Returns | |
---|---|
list_clipped
|
A list of Tensors of the same type as list_t .
|
global_norm
|
A 0-D (scalar) Tensor representing the global norm.
|
Raises | |
---|---|
TypeError
|
If t_list is not a sequence.
|
References:
On the difficulty of training Recurrent Neural Networks: Pascanu et al., 2012 (pdf)