 View on TensorFlow.org |
 Run in Google Colab |
 View source on GitHub |
 Download notebook |
概述
此笔记本将简要介绍 TensorFlow 的归一化层。当前支持的层包括：
- 组归一化（TensorFlow Addons）
- 实例归一化（TensorFlow Addons）
- 层归一化（TensorFlow Core）
这些层背后的基本理念是对激活层的输出进行归一化，以提升训练过程中的收敛。与批次归一化相反，这些归一化不适用于批次，而是用于归一化单个样本的激活，这样可使它们同样适用于循环神经网络。
通常，通过计算输入张量中子组的均值和标准差来执行归一化。此外，也可以对此应用比例因子和修正因子。
\(y_{i} = \frac{\gamma ( x_{i} - \mu )}{\sigma }+ \beta\)
\( y\)：输出
\(x\)：输入
\(\gamma\)：比例因子
\(\mu\)：均值
\(\sigma\)：标准差
\(\beta\)：修正因子
下面的图像演示了这些技术之间的区别。每个子图显示一个输入张量，其中 N 为批次轴，C 为通道轴，(H, W) 为空间轴（例如图片的高度和宽度）。蓝色像素由相同的均值和方差归一化，均值和方差通过聚合这些像素的值得出。
来源：(https://arxiv.org/pdf/1803.08494.pdf)
权重 γ 和 β 可以在所有归一化层中训练，以补偿表征能力的可能损失。您可以通过将 center 或 scale 标记设置为 True 来激活这些因子。当然，您也可以在训练过程中对 beta 和 gamma 使用 initializers、constraints 和 regularizer 来调整这些值。
设置
安装 Tensorflow 2.0 和 Tensorflow-Addons
pip install -q -U tensorflow-addonsimport tensorflow as tf
import tensorflow_addons as tfa
准备数据集
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
8192/11490434 [..............................] - ETA: 0s
4202496/11490434 [=========>....................] - ETA: 0s
9838592/11490434 [========================>.....] - ETA: 0s
11493376/11490434 [==============================] - 0s 0us/step
组归一化教程
简介
组归一化 (GN) 将输入的通道分成较小的子组，并根据其均值和方差归一化这些值。由于 GN 只对单一样本起作用，因此该技术与批次大小无关。
在图像分类任务中，GN 的实验得分与批次归一化十分接近。如果您的整体 批次大小很小，则使用 GN 而不是批次归一化可能更为有利，因为较小的批次大小会导致批次归一化的性能不佳。
下面的示例在 Conv2D 层之后将 10 个通道按标准的“最后一个通道”设置分为 5 个子组：
model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# Groupnorm Layer
tfa.layers.GroupNormalization(groups=5, axis=3),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)
1/313 [..............................] - ETA: 0s - loss: 2.6802 - accuracy: 0.1562 23/313 [=>............................] - ETA: 0s - loss: 1.4086 - accuracy: 0.6223 45/313 [===>..........................] - ETA: 0s - loss: 1.0807 - accuracy: 0.7028 67/313 [=====>........................] - ETA: 0s - loss: 0.8995 - accuracy: 0.7430 90/313 [=======>......................] - ETA: 0s - loss: 0.7721 - accuracy: 0.7795 112/313 [=========>....................] - ETA: 0s - loss: 0.7078 - accuracy: 0.7938 135/313 [===========>..................] - ETA: 0s - loss: 0.6590 - accuracy: 0.8090 157/313 [==============>...............] - ETA: 0s - loss: 0.6255 - accuracy: 0.8173 179/313 [================>.............] - ETA: 0s - loss: 0.5991 - accuracy: 0.8272 202/313 [==================>...........] - ETA: 0s - loss: 0.5719 - accuracy: 0.8342 225/313 [====================>.........] - ETA: 0s - loss: 0.5554 - accuracy: 0.8399 248/313 [======================>.......] - ETA: 0s - loss: 0.5410 - accuracy: 0.8429 270/313 [========================>.....] - ETA: 0s - loss: 0.5217 - accuracy: 0.8479 293/313 [===========================>..] - ETA: 0s - loss: 0.5099 - accuracy: 0.8516 313/313 [==============================] - ETA: 0s - loss: 0.4978 - accuracy: 0.8549 313/313 [==============================] - 1s 2ms/step - loss: 0.4978 - accuracy: 0.8549 <tensorflow.python.keras.callbacks.History at 0x7f116cc009e8>
实例归一化教程
简介
实例归一化是组归一化的特例，其中组大小与通道大小（或轴大小）相同。
实验结果表明，当替换批次归一化时，实例归一化在样式迁移方面表现良好。最近，实例归一化也已被用来代替 GAN 中的批次归一化。
示例
在 Conv2D 层之后应用 InstanceNormalization 并使用统一的初始化比例和偏移因子。
model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tfa.layers.InstanceNormalization(axis=3,
center=True,
scale=True,
beta_initializer="random_uniform",
gamma_initializer="random_uniform"),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)
1/313 [..............................] - ETA: 0s - loss: 2.3093 - accuracy: 0.1562 24/313 [=>............................] - ETA: 0s - loss: 2.0008 - accuracy: 0.3424 47/313 [===>..........................] - ETA: 0s - loss: 1.5448 - accuracy: 0.5279 70/313 [=====>........................] - ETA: 0s - loss: 1.2671 - accuracy: 0.6085 93/313 [=======>......................] - ETA: 0s - loss: 1.0761 - accuracy: 0.6687 116/313 [==========>...................] - ETA: 0s - loss: 0.9631 - accuracy: 0.7039 139/313 [============>.................] - ETA: 0s - loss: 0.8813 - accuracy: 0.7284 161/313 [==============>...............] - ETA: 0s - loss: 0.8085 - accuracy: 0.7510 184/313 [================>.............] - ETA: 0s - loss: 0.7536 - accuracy: 0.7677 206/313 [==================>...........] - ETA: 0s - loss: 0.7065 - accuracy: 0.7816 228/313 [====================>.........] - ETA: 0s - loss: 0.6685 - accuracy: 0.7930 251/313 [=======================>......] - ETA: 0s - loss: 0.6418 - accuracy: 0.8019 273/313 [=========================>....] - ETA: 0s - loss: 0.6147 - accuracy: 0.8110 296/313 [===========================>..] - ETA: 0s - loss: 0.5890 - accuracy: 0.8193 313/313 [==============================] - 1s 2ms/step - loss: 0.5717 - accuracy: 0.8241 <tensorflow.python.keras.callbacks.History at 0x7f116c97ae80>
层归一化教程
简介
层归一化是组归一化的特例，其中组大小为 1。均值和标准差根据单个样本的所有激活计算得出。
实验结果表明，层归一化非常适合循环神经网络，因为它可以独立于批大小工作。
示例
在 Conv2D 层之后应用 Layernormalization 并使用比例和偏移因子。
model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tf.keras.layers.LayerNormalization(axis=1 , center=True , scale=True),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)
1/313 [..............................] - ETA: 0s - loss: 2.8231 - accuracy: 0.0625 23/313 [=>............................] - ETA: 0s - loss: 0.9555 - accuracy: 0.6997 45/313 [===>..........................] - ETA: 0s - loss: 0.7968 - accuracy: 0.7660 67/313 [=====>........................] - ETA: 0s - loss: 0.7102 - accuracy: 0.7980 89/313 [=======>......................] - ETA: 0s - loss: 0.6493 - accuracy: 0.8136 110/313 [=========>....................] - ETA: 0s - loss: 0.5853 - accuracy: 0.8352 132/313 [===========>..................] - ETA: 0s - loss: 0.5574 - accuracy: 0.8452 153/313 [=============>................] - ETA: 0s - loss: 0.5373 - accuracy: 0.8517 175/313 [===============>..............] - ETA: 0s - loss: 0.5145 - accuracy: 0.8580 197/313 [=================>............] - ETA: 0s - loss: 0.4949 - accuracy: 0.8626 219/313 [===================>..........] - ETA: 0s - loss: 0.4734 - accuracy: 0.8674 241/313 [======================>.......] - ETA: 0s - loss: 0.4670 - accuracy: 0.8696 263/313 [========================>.....] - ETA: 0s - loss: 0.4527 - accuracy: 0.8723 285/313 [==========================>...] - ETA: 0s - loss: 0.4388 - accuracy: 0.8760 308/313 [============================>.] - ETA: 0s - loss: 0.4276 - accuracy: 0.8789 313/313 [==============================] - 1s 2ms/step - loss: 0.4242 - accuracy: 0.8794 <tensorflow.python.keras.callbacks.History at 0x7f116c7c1860>