# 归一化

## 概述

• 层归一化（TensorFlow Core）

$$y_{i} = \frac{\gamma ( x_{i} - \mu )}{\sigma }+ \beta$$

$$y$$：输出

$$x$$：输入

$$\gamma$$：比例因子

$$\mu$$：均值

$$\sigma$$：标准差

$$\beta$$：修正因子

## 设置

### 安装 Tensorflow 2.0 和 Tensorflow-Addons

pip install -q -U tensorflow-addons

import tensorflow as tf


### 准备数据集

mnist = tf.keras.datasets.mnist

x_train, x_test = x_train / 255.0, x_test / 255.0

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step


## 组归一化教程

### 下面的示例在 Conv2D 层之后将 10 个通道按标准的“最后一个通道”设置分为 5 个子组：

model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# Groupnorm Layer
tfa.layers.GroupNormalization(groups=5, axis=3),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])

loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)

313/313 [==============================] - 1s 2ms/step - loss: 0.4978 - accuracy: 0.8549
<tensorflow.python.keras.callbacks.History at 0x7f116cc009e8>


## 实例归一化教程

### 示例

model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tfa.layers.InstanceNormalization(axis=3,
center=True,
scale=True,
beta_initializer="random_uniform",
gamma_initializer="random_uniform"),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])

loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)

313/313 [==============================] - 1s 2ms/step - loss: 0.5717 - accuracy: 0.8241
<tensorflow.python.keras.callbacks.History at 0x7f116c97ae80>


## 层归一化教程

### 示例

model = tf.keras.models.Sequential([
# Reshape into "channels last" setup.
tf.keras.layers.Reshape((28,28,1), input_shape=(28,28)),
tf.keras.layers.Conv2D(filters=10, kernel_size=(3,3),data_format="channels_last"),
# LayerNorm Layer
tf.keras.layers.LayerNormalization(axis=1 , center=True , scale=True),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')
])

loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_test, y_test)

313/313 [==============================] - 1s 2ms/step - loss: 0.4242 - accuracy: 0.8794
<tensorflow.python.keras.callbacks.History at 0x7f116c7c1860>


## 文献

Layer norm

Instance norm

Group Norm

Complete Normalizations Overview

[{ "type": "thumb-down", "id": "missingTheInformationINeed", "label":"没有我需要的信息" },{ "type": "thumb-down", "id": "tooComplicatedTooManySteps", "label":"太复杂/步骤太多" },{ "type": "thumb-down", "id": "outOfDate", "label":"内容需要更新" },{ "type": "thumb-down", "id": "translationIssue", "label":"翻译问题" },{ "type": "thumb-down", "id": "samplesCodeIssue", "label":"示例/代码问题" },{ "type": "thumb-down", "id": "otherDown", "label":"其他" }]
[{ "type": "thumb-up", "id": "easyToUnderstand", "label":"易于理解" },{ "type": "thumb-up", "id": "solvedMyProblem", "label":"解决了我的问题" },{ "type": "thumb-up", "id": "otherUp", "label":"其他" }]