 在 TensorFlow.org 上查看 |
 在 Google Colab 中运行 |
 在 GitHub 上查看源代码 |
 下载笔记本 |
概述
此笔记本将演示如何使用 Addons 软件包中的条件梯度优化器。
ConditionalGradient
由于潜在的正则化效果，约束神经网络的参数已被证明对训练有益。通常，参数通过软惩罚（从不保证约束满足）或通过投影运算（计算资源消耗大）进行约束。另一方面，条件梯度 (CG) 优化器可严格执行约束，而无需消耗资源的投影步骤。它通过最大程度减小约束集中目标的线性逼近来工作。在此笔记本中，我们通过 MNIST 数据集上的 CG 优化器演示弗罗宾尼斯范数约束的应用。CG 现在可以作为 Tensorflow API 提供。有关优化器的更多详细信息，请参阅 https://arxiv.org/pdf/1803.06453.pdf
设置
pip install -U tensorflow-addonsimport tensorflow as tf
import tensorflow_addons as tfa
from matplotlib import pyplot as plt
# Hyperparameters
batch_size=64
epochs=10
构建模型
model_1 = tf.keras.Sequential([
tf.keras.layers.Dense(64, input_shape=(784,), activation='relu', name='dense_1'),
tf.keras.layers.Dense(64, activation='relu', name='dense_2'),
tf.keras.layers.Dense(10, activation='softmax', name='predictions'),
])
准备数据
# Load MNIST dataset as NumPy arrays
dataset = {}
num_validation = 10000
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
# Preprocess the data
x_train = x_train.reshape(-1, 784).astype('float32') / 255
x_test = x_test.reshape(-1, 784).astype('float32') / 255
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz 16384/11490434 [..............................] - ETA: 0s 5996544/11490434 [==============>...............] - ETA: 0s 11493376/11490434 [==============================] - 0s 0us/step 11501568/11490434 [==============================] - 0s 0us/step
定义自定义回调函数
def frobenius_norm(m):
"""This function is to calculate the frobenius norm of the matrix of all
layer's weight.
Args:
m: is a list of weights param for each layers.
"""
total_reduce_sum = 0
for i in range(len(m)):
total_reduce_sum = total_reduce_sum + tf.math.reduce_sum(m[i]**2)
norm = total_reduce_sum**0.5
return norm
CG_frobenius_norm_of_weight = []
CG_get_weight_norm = tf.keras.callbacks.LambdaCallback(
on_epoch_end=lambda batch, logs: CG_frobenius_norm_of_weight.append(
frobenius_norm(model_1.trainable_weights).numpy()))
训练和评估：使用 CG 作为优化器
只需用新的 TFA 优化器替换典型的 Keras 优化器
# Compile the model
model_1.compile(
optimizer=tfa.optimizers.ConditionalGradient(
learning_rate=0.99949, lambda_=203), # Utilize TFA optimizer
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
metrics=['accuracy'])
history_cg = model_1.fit(
x_train,
y_train,
batch_size=batch_size,
validation_data=(x_test, y_test),
epochs=epochs,
callbacks=[CG_get_weight_norm])
Epoch 1/10 1/938 [..............................] - ETA: 12:44 - loss: 2.3203 - accuracy: 0.1094 22/938 [..............................] - ETA: 2s - loss: 1.7988 - accuracy: 0.4375 43/938 [>.............................] - ETA: 2s - loss: 1.4224 - accuracy: 0.5672 64/938 [=>............................] - ETA: 2s - loss: 1.1994 - accuracy: 0.6365 85/938 [=>............................] - ETA: 2s - loss: 1.0505 - accuracy: 0.6827 106/938 [==>...........................] - ETA: 2s - loss: 0.9615 - accuracy: 0.7109 127/938 [===>..........................] - ETA: 1s - loss: 0.8899 - accuracy: 0.7331 148/938 [===>..........................] - ETA: 1s - loss: 0.8228 - accuracy: 0.7538 169/938 [====>.........................] - ETA: 1s - loss: 0.7745 - accuracy: 0.7683 189/938 [=====>........................] - ETA: 1s - loss: 0.7370 - accuracy: 0.7794 208/938 [=====>........................] - ETA: 1s - loss: 0.7102 - accuracy: 0.7873 229/938 [======>.......................] - ETA: 1s - loss: 0.6827 - accuracy: 0.7952 250/938 [======>.......................] - ETA: 1s - loss: 0.6539 - accuracy: 0.8044 271/938 [=======>......................] - ETA: 1s - loss: 0.6312 - accuracy: 0.8113 291/938 [========>.....................] - ETA: 1s - loss: 0.6147 - accuracy: 0.8163 313/938 [=========>....................] - ETA: 1s - loss: 0.5976 - accuracy: 0.8212 335/938 [=========>....................] - ETA: 1s - loss: 0.5812 - accuracy: 0.8259 357/938 [==========>...................] - ETA: 1s - loss: 0.5670 - accuracy: 0.8297 379/938 [===========>..................] - ETA: 1s - loss: 0.5552 - accuracy: 0.8332 400/938 [===========>..................] - ETA: 1s - loss: 0.5448 - accuracy: 0.8368 422/938 [============>.................] - ETA: 1s - loss: 0.5334 - accuracy: 0.8400 443/938 [=============>................] - ETA: 1s - loss: 0.5240 - accuracy: 0.8430 466/938 [=============>................] - ETA: 1s - loss: 0.5127 - accuracy: 0.8459 488/938 [==============>...............] - ETA: 1s - loss: 0.5030 - accuracy: 0.8487 511/938 [===============>..............] - ETA: 1s - loss: 0.4931 - accuracy: 0.8514 534/938 [================>.............] - ETA: 0s - loss: 0.4847 - accuracy: 0.8538 556/938 [================>.............] - ETA: 0s - loss: 0.4797 - accuracy: 0.8553 579/938 [=================>............] - ETA: 0s - loss: 0.4719 - accuracy: 0.8577 601/938 [==================>...........] - ETA: 0s - loss: 0.4647 - accuracy: 0.8600 622/938 [==================>...........] - ETA: 0s - loss: 0.4582 - accuracy: 0.8617 645/938 [===================>..........] - ETA: 0s - loss: 0.4522 - accuracy: 0.8640 667/938 [====================>.........] - ETA: 0s - loss: 0.4460 - accuracy: 0.8656 689/938 [=====================>........] - ETA: 0s - loss: 0.4403 - accuracy: 0.8672 712/938 [=====================>........] - ETA: 0s - loss: 0.4344 - accuracy: 0.8692 735/938 [======================>.......] - ETA: 0s - loss: 0.4290 - accuracy: 0.8707 758/938 [=======================>......] - ETA: 0s - loss: 0.4240 - accuracy: 0.8723 781/938 [=======================>......] - ETA: 0s - loss: 0.4191 - accuracy: 0.8738 804/938 [========================>.....] - ETA: 0s - loss: 0.4138 - accuracy: 0.8754 827/938 [=========================>....] - ETA: 0s - loss: 0.4088 - accuracy: 0.8769 849/938 [==========================>...] - ETA: 0s - loss: 0.4044 - accuracy: 0.8782 872/938 [==========================>...] - ETA: 0s - loss: 0.4007 - accuracy: 0.8795 894/938 [===========================>..] - ETA: 0s - loss: 0.3971 - accuracy: 0.8806 917/938 [============================>.] - ETA: 0s - loss: 0.3930 - accuracy: 0.8819 938/938 [==============================] - ETA: 0s - loss: 0.3901 - accuracy: 0.8826 938/938 [==============================] - 3s 3ms/step - loss: 0.3901 - accuracy: 0.8826 - val_loss: 0.2070 - val_accuracy: 0.9375 Epoch 2/10 1/938 [..............................] - ETA: 2s - loss: 0.1675 - accuracy: 0.9375 22/938 [..............................] - ETA: 2s - loss: 0.2277 - accuracy: 0.9318 43/938 [>.............................] - ETA: 2s - loss: 0.2202 - accuracy: 0.9390 64/938 [=>............................] - ETA: 2s - loss: 0.2283 - accuracy: 0.9329 86/938 [=>............................] - ETA: 2s - loss: 0.2287 - accuracy: 0.9337 109/938 [==>...........................] - ETA: 1s - loss: 0.2253 - accuracy: 0.9332 132/938 [===>..........................] - ETA: 1s - loss: 0.2236 - accuracy: 0.9336 154/938 [===>..........................] - ETA: 1s - loss: 0.2208 - accuracy: 0.9352 177/938 [====>.........................] - ETA: 1s - loss: 0.2163 - accuracy: 0.9362 199/938 [=====>........................] - ETA: 1s - loss: 0.2156 - accuracy: 0.9359 220/938 [======>.......................] - ETA: 1s - loss: 0.2132 - accuracy: 0.9362 242/938 [======>.......................] - ETA: 1s - loss: 0.2127 - accuracy: 0.9370 263/938 [=======>......................] - ETA: 1s - loss: 0.2106 - accuracy: 0.9370 285/938 [========>.....................] - ETA: 1s - loss: 0.2103 - accuracy: 0.9376 307/938 [========>.....................] - ETA: 1s - loss: 0.2088 - accuracy: 0.9383 329/938 [=========>....................] - ETA: 1s - loss: 0.2073 - accuracy: 0.9383 351/938 [==========>...................] - ETA: 1s - loss: 0.2066 - accuracy: 0.9387 371/938 [==========>...................] - ETA: 1s - loss: 0.2040 - accuracy: 0.9396 392/938 [===========>..................] - ETA: 1s - loss: 0.2038 - accuracy: 0.9398 413/938 [============>.................] - ETA: 1s - loss: 0.2027 - accuracy: 0.9405 436/938 [============>.................] - ETA: 1s - loss: 0.2018 - accuracy: 0.9407 458/938 [=============>................] - ETA: 1s - loss: 0.2010 - accuracy: 0.9407 481/938 [==============>...............] - ETA: 1s - loss: 0.2011 - accuracy: 0.9406 504/938 [===============>..............] - ETA: 1s - loss: 0.2021 - accuracy: 0.9403 526/938 [===============>..............] - ETA: 0s - loss: 0.2008 - accuracy: 0.9405 548/938 [================>.............] - ETA: 0s - loss: 0.2005 - accuracy: 0.9405 570/938 [=================>............] - ETA: 0s - loss: 0.2001 - accuracy: 0.9405 592/938 [=================>............] - ETA: 0s - loss: 0.1989 - accuracy: 0.9406 614/938 [==================>...........] - ETA: 0s - loss: 0.1973 - accuracy: 0.9411 636/938 [===================>..........] - ETA: 0s - loss: 0.1970 - accuracy: 0.9413 658/938 [====================>.........] - ETA: 0s - loss: 0.1961 - accuracy: 0.9415 681/938 [====================>.........] - ETA: 0s - loss: 0.1959 - accuracy: 0.9417 704/938 [=====================>........] - ETA: 0s - loss: 0.1948 - accuracy: 0.9419 726/938 [======================>.......] - ETA: 0s - loss: 0.1944 - accuracy: 0.9419 748/938 [======================>.......] - ETA: 0s - loss: 0.1931 - accuracy: 0.9424 771/938 [=======================>......] - ETA: 0s - loss: 0.1927 - accuracy: 0.9426 793/938 [========================>.....] - ETA: 0s - loss: 0.1918 - accuracy: 0.9429 816/938 [=========================>....] - ETA: 0s - loss: 0.1911 - accuracy: 0.9429 837/938 [=========================>....] - ETA: 0s - loss: 0.1901 - accuracy: 0.9432 858/938 [==========================>...] - ETA: 0s - loss: 0.1895 - accuracy: 0.9433 880/938 [===========================>..] - ETA: 0s - loss: 0.1892 - accuracy: 0.9434 902/938 [===========================>..] - ETA: 0s - loss: 0.1889 - accuracy: 0.9436 924/938 [============================>.] - ETA: 0s - loss: 0.1880 - accuracy: 0.9440 938/938 [==============================] - 2s 3ms/step - loss: 0.1871 - accuracy: 0.9442 - val_loss: 0.1567 - val_accuracy: 0.9507 Epoch 3/10 1/938 [..............................] - ETA: 2s - loss: 0.0778 - accuracy: 0.9688 23/938 [..............................] - ETA: 2s - loss: 0.1878 - accuracy: 0.9429 45/938 [>.............................] - ETA: 2s - loss: 0.1738 - accuracy: 0.9455 67/938 [=>............................] - ETA: 2s - loss: 0.1745 - accuracy: 0.9468 90/938 [=>............................] - ETA: 1s - loss: 0.1661 - accuracy: 0.9507 112/938 [==>...........................] - ETA: 1s - loss: 0.1643 - accuracy: 0.9501 134/938 [===>..........................] - ETA: 1s - loss: 0.1671 - accuracy: 0.9497 155/938 [===>..........................] - ETA: 1s - loss: 0.1657 - accuracy: 0.9511 177/938 [====>.........................] - ETA: 1s - loss: 0.1634 - accuracy: 0.9515 198/938 [=====>........................] - ETA: 1s - loss: 0.1617 - accuracy: 0.9515 219/938 [======>.......................] - ETA: 1s - loss: 0.1598 - accuracy: 0.9523 241/938 [======>.......................] - ETA: 1s - loss: 0.1610 - accuracy: 0.9513 263/938 [=======>......................] - ETA: 1s - loss: 0.1604 - accuracy: 0.9518 285/938 [========>.....................] - ETA: 1s - loss: 0.1586 - accuracy: 0.9526 307/938 [========>.....................] - ETA: 1s - loss: 0.1568 - accuracy: 0.9533 329/938 [=========>....................] - ETA: 1s - loss: 0.1548 - accuracy: 0.9542 352/938 [==========>...................] - ETA: 1s - loss: 0.1556 - accuracy: 0.9545 374/938 [==========>...................] - ETA: 1s - loss: 0.1544 - accuracy: 0.9547 396/938 [===========>..................] - ETA: 1s - loss: 0.1541 - accuracy: 0.9546 418/938 [============>.................] - ETA: 1s - loss: 0.1531 - accuracy: 0.9547 440/938 [=============>................] - ETA: 1s - loss: 0.1524 - accuracy: 0.9549 461/938 [=============>................] - ETA: 1s - loss: 0.1528 - accuracy: 0.9551 482/938 [==============>...............] - ETA: 1s - loss: 0.1519 - accuracy: 0.9552 504/938 [===============>..............] - ETA: 1s - loss: 0.1521 - accuracy: 0.9555 526/938 [===============>..............] - ETA: 0s - loss: 0.1511 - accuracy: 0.9556 548/938 [================>.............] - ETA: 0s - loss: 0.1500 - accuracy: 0.9561 570/938 [=================>............] - ETA: 0s - loss: 0.1495 - accuracy: 0.9562 592/938 [=================>............] - ETA: 0s - loss: 0.1491 - accuracy: 0.9562 612/938 [==================>...........] - ETA: 0s - loss: 0.1491 - accuracy: 0.9562 633/938 [===================>..........] - ETA: 0s - loss: 0.1493 - accuracy: 0.9561 653/938 [===================>..........] - ETA: 0s - loss: 0.1489 - accuracy: 0.9561 674/938 [====================>.........] - ETA: 0s - loss: 0.1490 - accuracy: 0.9560 694/938 [=====================>........] - ETA: 0s - loss: 0.1488 - accuracy: 0.9558 715/938 [=====================>........] - ETA: 0s - loss: 0.1489 - accuracy: 0.9558 737/938 [======================>.......] - ETA: 0s - loss: 0.1485 - accuracy: 0.9560 760/938 [=======================>......] - ETA: 0s - loss: 0.1483 - accuracy: 0.9563 781/938 [=======================>......] - ETA: 0s - loss: 0.1484 - accuracy: 0.9562 802/938 [========================>.....] - ETA: 0s - loss: 0.1479 - accuracy: 0.9564 824/938 [=========================>....] - ETA: 0s - loss: 0.1476 - accuracy: 0.9563 846/938 [==========================>...] - ETA: 0s - loss: 0.1477 - accuracy: 0.9564 868/938 [==========================>...] - ETA: 0s - loss: 0.1473 - accuracy: 0.9564 891/938 [===========================>..] - ETA: 0s - loss: 0.1481 - accuracy: 0.9561 912/938 [============================>.] - ETA: 0s - loss: 0.1475 - accuracy: 0.9563 933/938 [============================>.] - ETA: 0s - loss: 0.1471 - accuracy: 0.9563 938/938 [==============================] - 2s 3ms/step - loss: 0.1471 - accuracy: 0.9561 - val_loss: 0.1253 - val_accuracy: 0.9635 Epoch 4/10 1/938 [..............................] - ETA: 2s - loss: 0.0675 - accuracy: 0.9844 23/938 [..............................] - ETA: 2s - loss: 0.1277 - accuracy: 0.9626 45/938 [>.............................] - ETA: 2s - loss: 0.1220 - accuracy: 0.9625 67/938 [=>............................] - ETA: 2s - loss: 0.1252 - accuracy: 0.9618 89/938 [=>............................] - ETA: 1s - loss: 0.1222 - accuracy: 0.9630 111/938 [==>...........................] - ETA: 1s - loss: 0.1198 - accuracy: 0.9641 133/938 [===>..........................] - ETA: 1s - loss: 0.1199 - accuracy: 0.9635 156/938 [===>..........................] - ETA: 1s - loss: 0.1216 - accuracy: 0.9627 177/938 [====>.........................] - ETA: 1s - loss: 0.1228 - accuracy: 0.9617 198/938 [=====>........................] - ETA: 1s - loss: 0.1224 - accuracy: 0.9616 220/938 [======>.......................] - ETA: 1s - loss: 0.1249 - accuracy: 0.9611 242/938 [======>.......................] - ETA: 1s - loss: 0.1240 - accuracy: 0.9615 263/938 [=======>......................] - ETA: 1s - loss: 0.1242 - accuracy: 0.9615 285/938 [========>.....................] - ETA: 1s - loss: 0.1252 - accuracy: 0.9614 307/938 [========>.....................] - ETA: 1s - loss: 0.1250 - accuracy: 0.9617 330/938 [=========>....................] - ETA: 1s - loss: 0.1273 - accuracy: 0.9614 352/938 [==========>...................] - ETA: 1s - loss: 0.1273 - accuracy: 0.9614 374/938 [==========>...................] - ETA: 1s - loss: 0.1280 - accuracy: 0.9610 396/938 [===========>..................] - ETA: 1s - loss: 0.1284 - accuracy: 0.9609 418/938 [============>.................] - ETA: 1s - loss: 0.1284 - accuracy: 0.9607 440/938 [=============>................] - ETA: 1s - loss: 0.1293 - accuracy: 0.9607 462/938 [=============>................] - ETA: 1s - loss: 0.1286 - accuracy: 0.9609 484/938 [==============>...............] - ETA: 1s - loss: 0.1294 - accuracy: 0.9611 506/938 [===============>..............] - ETA: 1s - loss: 0.1285 - accuracy: 0.9615 528/938 [===============>..............] - ETA: 0s - loss: 0.1278 - accuracy: 0.9618 550/938 [================>.............] - ETA: 0s - loss: 0.1278 - accuracy: 0.9617 573/938 [=================>............] - ETA: 0s - loss: 0.1275 - accuracy: 0.9616 596/938 [==================>...........] - ETA: 0s - loss: 0.1281 - accuracy: 0.9614 618/938 [==================>...........] - ETA: 0s - loss: 0.1279 - accuracy: 0.9614 640/938 [===================>..........] - ETA: 0s - loss: 0.1286 - accuracy: 0.9611 663/938 [====================>.........] - ETA: 0s - loss: 0.1284 - accuracy: 0.9612 684/938 [====================>.........] - ETA: 0s - loss: 0.1288 - accuracy: 0.9611 706/938 [=====================>........] - ETA: 0s - loss: 0.1284 - accuracy: 0.9613 727/938 [======================>.......] - ETA: 0s - loss: 0.1282 - accuracy: 0.9613 749/938 [======================>.......] - ETA: 0s - loss: 0.1274 - accuracy: 0.9615 772/938 [=======================>......] - ETA: 0s - loss: 0.1276 - accuracy: 0.9615 795/938 [========================>.....] - ETA: 0s - loss: 0.1277 - accuracy: 0.9615 818/938 [=========================>....] - ETA: 0s - loss: 0.1280 - accuracy: 0.9615 840/938 [=========================>....] - ETA: 0s - loss: 0.1282 - accuracy: 0.9614 862/938 [==========================>...] - ETA: 0s - loss: 0.1279 - accuracy: 0.9616 884/938 [===========================>..] - ETA: 0s - loss: 0.1281 - accuracy: 0.9615 906/938 [===========================>..] - ETA: 0s - loss: 0.1283 - accuracy: 0.9615 928/938 [============================>.] - ETA: 0s - loss: 0.1283 - accuracy: 0.9615 938/938 [==============================] - 2s 3ms/step - loss: 0.1282 - accuracy: 0.9616 - val_loss: 0.1526 - val_accuracy: 0.9493 Epoch 5/10 1/938 [..............................] - ETA: 2s - loss: 0.0520 - accuracy: 0.9844 24/938 [..............................] - ETA: 2s - loss: 0.0993 - accuracy: 0.9681 46/938 [>.............................] - ETA: 2s - loss: 0.0987 - accuracy: 0.9704 68/938 [=>............................] - ETA: 1s - loss: 0.1065 - accuracy: 0.9665 90/938 [=>............................] - ETA: 1s - loss: 0.1100 - accuracy: 0.9660 112/938 [==>...........................] - ETA: 1s - loss: 0.1106 - accuracy: 0.9654 134/938 [===>..........................] - ETA: 1s - loss: 0.1092 - accuracy: 0.9664 157/938 [====>.........................] - ETA: 1s - loss: 0.1109 - accuracy: 0.9661 179/938 [====>.........................] - ETA: 1s - loss: 0.1137 - accuracy: 0.9647 201/938 [=====>........................] - ETA: 1s - loss: 0.1160 - accuracy: 0.9640 224/938 [======>.......................] - ETA: 1s - loss: 0.1149 - accuracy: 0.9644 246/938 [======>.......................] - ETA: 1s - loss: 0.1160 - accuracy: 0.9640 269/938 [=======>......................] - ETA: 1s - loss: 0.1154 - accuracy: 0.9639 291/938 [========>.....................] - ETA: 1s - loss: 0.1153 - accuracy: 0.9639 314/938 [=========>....................] - ETA: 1s - loss: 0.1154 - accuracy: 0.9639 336/938 [=========>....................] - ETA: 1s - loss: 0.1147 - accuracy: 0.9645 358/938 [==========>...................] - ETA: 1s - loss: 0.1149 - accuracy: 0.9645 380/938 [===========>..................] - ETA: 1s - loss: 0.1155 - accuracy: 0.9646 403/938 [===========>..................] - ETA: 1s - loss: 0.1159 - accuracy: 0.9644 425/938 [============>.................] - ETA: 1s - loss: 0.1166 - accuracy: 0.9643 447/938 [=============>................] - ETA: 1s - loss: 0.1172 - accuracy: 0.9641 468/938 [=============>................] - ETA: 1s - loss: 0.1174 - accuracy: 0.9638 490/938 [==============>...............] - ETA: 1s - loss: 0.1169 - accuracy: 0.9640 511/938 [===============>..............] - ETA: 0s - loss: 0.1167 - accuracy: 0.9642 533/938 [================>.............] - ETA: 0s - loss: 0.1170 - accuracy: 0.9641 554/938 [================>.............] - ETA: 0s - loss: 0.1169 - accuracy: 0.9643 576/938 [=================>............] - ETA: 0s - loss: 0.1168 - accuracy: 0.9645 598/938 [==================>...........] - ETA: 0s - loss: 0.1175 - accuracy: 0.9642 620/938 [==================>...........] - ETA: 0s - loss: 0.1182 - accuracy: 0.9638 642/938 [===================>..........] - ETA: 0s - loss: 0.1186 - accuracy: 0.9638 664/938 [====================>.........] - ETA: 0s - loss: 0.1181 - accuracy: 0.9639 686/938 [====================>.........] - ETA: 0s - loss: 0.1179 - accuracy: 0.9641 708/938 [=====================>........] - ETA: 0s - loss: 0.1185 - accuracy: 0.9639 730/938 [======================>.......] - ETA: 0s - loss: 0.1182 - accuracy: 0.9642 751/938 [=======================>......] - ETA: 0s - loss: 0.1178 - accuracy: 0.9644 773/938 [=======================>......] - ETA: 0s - loss: 0.1179 - accuracy: 0.9644 796/938 [========================>.....] - ETA: 0s - loss: 0.1183 - accuracy: 0.9642 818/938 [=========================>....] - ETA: 0s - loss: 0.1181 - accuracy: 0.9644 840/938 [=========================>....] - ETA: 0s - loss: 0.1192 - accuracy: 0.9640 862/938 [==========================>...] - ETA: 0s - loss: 0.1189 - accuracy: 0.9641 884/938 [===========================>..] - ETA: 0s - loss: 0.1187 - accuracy: 0.9642 907/938 [============================>.] - ETA: 0s - loss: 0.1183 - accuracy: 0.9644 930/938 [============================>.] - ETA: 0s - loss: 0.1181 - accuracy: 0.9644 938/938 [==============================] - 2s 3ms/step - loss: 0.1181 - accuracy: 0.9644 - val_loss: 0.2061 - val_accuracy: 0.9313 Epoch 6/10 1/938 [..............................] - ETA: 2s - loss: 0.1636 - accuracy: 0.9688 23/938 [..............................] - ETA: 2s - loss: 0.1092 - accuracy: 0.9633 45/938 [>.............................] - ETA: 2s - loss: 0.1077 - accuracy: 0.9628 67/938 [=>............................] - ETA: 2s - loss: 0.1104 - accuracy: 0.9629 89/938 [=>............................] - ETA: 1s - loss: 0.1121 - accuracy: 0.9635 111/938 [==>...........................] - ETA: 1s - loss: 0.1088 - accuracy: 0.9658 134/938 [===>..........................] - ETA: 1s - loss: 0.1077 - accuracy: 0.9658 156/938 [===>..........................] - ETA: 1s - loss: 0.1112 - accuracy: 0.9647 179/938 [====>.........................] - ETA: 1s - loss: 0.1115 - accuracy: 0.9653 202/938 [=====>........................] - ETA: 1s - loss: 0.1098 - accuracy: 0.9657 225/938 [======>.......................] - ETA: 1s - loss: 0.1105 - accuracy: 0.9650 247/938 [======>.......................] - ETA: 1s - loss: 0.1106 - accuracy: 0.9648 269/938 [=======>......................] - ETA: 1s - loss: 0.1092 - accuracy: 0.9657 291/938 [========>.....................] - ETA: 1s - loss: 0.1100 - accuracy: 0.9653 313/938 [=========>....................] - ETA: 1s - loss: 0.1107 - accuracy: 0.9655 335/938 [=========>....................] - ETA: 1s - loss: 0.1100 - accuracy: 0.9659 357/938 [==========>...................] - ETA: 1s - loss: 0.1088 - accuracy: 0.9664 379/938 [===========>..................] - ETA: 1s - loss: 0.1094 - accuracy: 0.9664 401/938 [===========>..................] - ETA: 1s - loss: 0.1097 - accuracy: 0.9665 423/938 [============>.................] - ETA: 1s - loss: 0.1100 - accuracy: 0.9663 445/938 [=============>................] - ETA: 1s - loss: 0.1104 - accuracy: 0.9665 467/938 [=============>................] - ETA: 1s - loss: 0.1100 - accuracy: 0.9664 489/938 [==============>...............] - ETA: 1s - loss: 0.1096 - accuracy: 0.9665 510/938 [===============>..............] - ETA: 0s - loss: 0.1099 - accuracy: 0.9666 533/938 [================>.............] - ETA: 0s - loss: 0.1094 - accuracy: 0.9666 554/938 [================>.............] - ETA: 0s - loss: 0.1096 - accuracy: 0.9666 577/938 [=================>............] - ETA: 0s - loss: 0.1101 - accuracy: 0.9667 599/938 [==================>...........] - ETA: 0s - loss: 0.1116 - accuracy: 0.9665 622/938 [==================>...........] - ETA: 0s - loss: 0.1112 - accuracy: 0.9666 643/938 [===================>..........] - ETA: 0s - loss: 0.1114 - accuracy: 0.9666 665/938 [====================>.........] - ETA: 0s - loss: 0.1117 - accuracy: 0.9665 687/938 [====================>.........] - ETA: 0s - loss: 0.1122 - accuracy: 0.9662 709/938 [=====================>........] - ETA: 0s - loss: 0.1128 - accuracy: 0.9660 731/938 [======================>.......] - ETA: 0s - loss: 0.1131 - accuracy: 0.9661 752/938 [=======================>......] - ETA: 0s - loss: 0.1130 - accuracy: 0.9661 773/938 [=======================>......] - ETA: 0s - loss: 0.1135 - accuracy: 0.9660 795/938 [========================>.....] - ETA: 0s - loss: 0.1138 - accuracy: 0.9659 818/938 [=========================>....] - ETA: 0s - loss: 0.1144 - accuracy: 0.9659 840/938 [=========================>....] - ETA: 0s - loss: 0.1143 - accuracy: 0.9660 862/938 [==========================>...] - ETA: 0s - loss: 0.1148 - accuracy: 0.9659 884/938 [===========================>..] - ETA: 0s - loss: 0.1148 - accuracy: 0.9659 906/938 [===========================>..] - ETA: 0s - loss: 0.1145 - accuracy: 0.9660 929/938 [============================>.] - ETA: 0s - loss: 0.1146 - accuracy: 0.9660 938/938 [==============================] - 2s 3ms/step - loss: 0.1147 - accuracy: 0.9659 - val_loss: 0.1480 - val_accuracy: 0.9559 Epoch 7/10 1/938 [..............................] - ETA: 2s - loss: 0.0738 - accuracy: 1.0000 23/938 [..............................] - ETA: 2s - loss: 0.1074 - accuracy: 0.9660 45/938 [>.............................] - ETA: 2s - loss: 0.1061 - accuracy: 0.9681 67/938 [=>............................] - ETA: 2s - loss: 0.1012 - accuracy: 0.9688 89/938 [=>............................] - ETA: 1s - loss: 0.1027 - accuracy: 0.9689 110/938 [==>...........................] - ETA: 1s - loss: 0.1036 - accuracy: 0.9686 132/938 [===>..........................] - ETA: 1s - loss: 0.1023 - accuracy: 0.9697 153/938 [===>..........................] - ETA: 1s - loss: 0.1023 - accuracy: 0.9698 176/938 [====>.........................] - ETA: 1s - loss: 0.1028 - accuracy: 0.9694 199/938 [=====>........................] - ETA: 1s - loss: 0.1015 - accuracy: 0.9697 222/938 [======>.......................] - ETA: 1s - loss: 0.1007 - accuracy: 0.9699 245/938 [======>.......................] - ETA: 1s - loss: 0.1009 - accuracy: 0.9696 267/938 [=======>......................] - ETA: 1s - loss: 0.1014 - accuracy: 0.9692 290/938 [========>.....................] - ETA: 1s - loss: 0.1032 - accuracy: 0.9681 312/938 [========>.....................] - ETA: 1s - loss: 0.1036 - accuracy: 0.9676 334/938 [=========>....................] - ETA: 1s - loss: 0.1042 - accuracy: 0.9676 356/938 [==========>...................] - ETA: 1s - loss: 0.1034 - accuracy: 0.9680 377/938 [===========>..................] - ETA: 1s - loss: 0.1030 - accuracy: 0.9683 400/938 [===========>..................] - ETA: 1s - loss: 0.1032 - accuracy: 0.9680 423/938 [============>.................] - ETA: 1s - loss: 0.1043 - accuracy: 0.9678 446/938 [=============>................] - ETA: 1s - loss: 0.1046 - accuracy: 0.9677 469/938 [==============>...............] - ETA: 1s - loss: 0.1048 - accuracy: 0.9676 492/938 [==============>...............] - ETA: 1s - loss: 0.1053 - accuracy: 0.9673 515/938 [===============>..............] - ETA: 0s - loss: 0.1059 - accuracy: 0.9674 538/938 [================>.............] - ETA: 0s - loss: 0.1066 - accuracy: 0.9671 560/938 [================>.............] - ETA: 0s - loss: 0.1063 - accuracy: 0.9672 581/938 [=================>............] - ETA: 0s - loss: 0.1065 - accuracy: 0.9672 603/938 [==================>...........] - ETA: 0s - loss: 0.1062 - accuracy: 0.9672 626/938 [===================>..........] - ETA: 0s - loss: 0.1070 - accuracy: 0.9671 647/938 [===================>..........] - ETA: 0s - loss: 0.1072 - accuracy: 0.9671 669/938 [====================>.........] - ETA: 0s - loss: 0.1075 - accuracy: 0.9671 692/938 [=====================>........] - ETA: 0s - loss: 0.1076 - accuracy: 0.9671 714/938 [=====================>........] - ETA: 0s - loss: 0.1077 - accuracy: 0.9670 736/938 [======================>.......] - ETA: 0s - loss: 0.1077 - accuracy: 0.9671 759/938 [=======================>......] - ETA: 0s - loss: 0.1079 - accuracy: 0.9670 783/938 [========================>.....] - ETA: 0s - loss: 0.1085 - accuracy: 0.9667 805/938 [========================>.....] - ETA: 0s - loss: 0.1089 - accuracy: 0.9666 828/938 [=========================>....] - ETA: 0s - loss: 0.1092 - accuracy: 0.9665 851/938 [==========================>...] - ETA: 0s - loss: 0.1092 - accuracy: 0.9665 873/938 [==========================>...] - ETA: 0s - loss: 0.1092 - accuracy: 0.9664 895/938 [===========================>..] - ETA: 0s - loss: 0.1090 - accuracy: 0.9666 918/938 [============================>.] - ETA: 0s - loss: 0.1085 - accuracy: 0.9668 938/938 [==============================] - 2s 3ms/step - loss: 0.1084 - accuracy: 0.9668 - val_loss: 0.1183 - val_accuracy: 0.9663 Epoch 8/10 1/938 [..............................] - ETA: 2s - loss: 0.0599 - accuracy: 0.9844 23/938 [..............................] - ETA: 2s - loss: 0.0878 - accuracy: 0.9755 45/938 [>.............................] - ETA: 2s - loss: 0.1012 - accuracy: 0.9701 68/938 [=>............................] - ETA: 1s - loss: 0.1022 - accuracy: 0.9701 91/938 [=>............................] - ETA: 1s - loss: 0.0982 - accuracy: 0.9705 113/938 [==>...........................] - ETA: 1s - loss: 0.0971 - accuracy: 0.9708 136/938 [===>..........................] - ETA: 1s - loss: 0.0964 - accuracy: 0.9701 159/938 [====>.........................] - ETA: 1s - loss: 0.0962 - accuracy: 0.9701 182/938 [====>.........................] - ETA: 1s - loss: 0.0977 - accuracy: 0.9698 204/938 [=====>........................] - ETA: 1s - loss: 0.0971 - accuracy: 0.9702 227/938 [======>.......................] - ETA: 1s - loss: 0.0989 - accuracy: 0.9699 250/938 [======>.......................] - ETA: 1s - loss: 0.0984 - accuracy: 0.9703 273/938 [=======>......................] - ETA: 1s - loss: 0.0991 - accuracy: 0.9701 296/938 [========>.....................] - ETA: 1s - loss: 0.0988 - accuracy: 0.9705 319/938 [=========>....................] - ETA: 1s - loss: 0.0996 - accuracy: 0.9700 342/938 [=========>....................] - ETA: 1s - loss: 0.1008 - accuracy: 0.9694 365/938 [==========>...................] - ETA: 1s - loss: 0.1002 - accuracy: 0.9695 387/938 [===========>..................] - ETA: 1s - loss: 0.0999 - accuracy: 0.9694 410/938 [============>.................] - ETA: 1s - loss: 0.1004 - accuracy: 0.9690 433/938 [============>.................] - ETA: 1s - loss: 0.0997 - accuracy: 0.9695 456/938 [=============>................] - ETA: 1s - loss: 0.0997 - accuracy: 0.9695 479/938 [==============>...............] - ETA: 1s - loss: 0.1006 - accuracy: 0.9694 502/938 [===============>..............] - ETA: 0s - loss: 0.1012 - accuracy: 0.9690 526/938 [===============>..............] - ETA: 0s - loss: 0.1008 - accuracy: 0.9692 550/938 [================>.............] - ETA: 0s - loss: 0.1012 - accuracy: 0.9691 573/938 [=================>............] - ETA: 0s - loss: 0.1012 - accuracy: 0.9693 596/938 [==================>...........] - ETA: 0s - loss: 0.1013 - accuracy: 0.9692 619/938 [==================>...........] - ETA: 0s - loss: 0.1013 - accuracy: 0.9692 642/938 [===================>..........] - ETA: 0s - loss: 0.1012 - accuracy: 0.9692 665/938 [====================>.........] - ETA: 0s - loss: 0.1018 - accuracy: 0.9693 688/938 [=====================>........] - ETA: 0s - loss: 0.1018 - accuracy: 0.9690 711/938 [=====================>........] - ETA: 0s - loss: 0.1014 - accuracy: 0.9693 735/938 [======================>.......] - ETA: 0s - loss: 0.1026 - accuracy: 0.9689 757/938 [=======================>......] - ETA: 0s - loss: 0.1027 - accuracy: 0.9689 780/938 [=======================>......] - ETA: 0s - loss: 0.1029 - accuracy: 0.9688 802/938 [========================>.....] - ETA: 0s - loss: 0.1033 - accuracy: 0.9687 824/938 [=========================>....] - ETA: 0s - loss: 0.1040 - accuracy: 0.9685 846/938 [==========================>...] - ETA: 0s - loss: 0.1043 - accuracy: 0.9684 868/938 [==========================>...] - ETA: 0s - loss: 0.1042 - accuracy: 0.9685 890/938 [===========================>..] - ETA: 0s - loss: 0.1045 - accuracy: 0.9684 912/938 [============================>.] - ETA: 0s - loss: 0.1046 - accuracy: 0.9683 935/938 [============================>.] - ETA: 0s - loss: 0.1049 - accuracy: 0.9683 938/938 [==============================] - 2s 3ms/step - loss: 0.1051 - accuracy: 0.9682 - val_loss: 0.1599 - val_accuracy: 0.9471 Epoch 9/10 1/938 [..............................] - ETA: 2s - loss: 0.2096 - accuracy: 0.9375 24/938 [..............................] - ETA: 2s - loss: 0.1139 - accuracy: 0.9635 46/938 [>.............................] - ETA: 2s - loss: 0.1103 - accuracy: 0.9637 68/938 [=>............................] - ETA: 2s - loss: 0.1073 - accuracy: 0.9658 90/938 [=>............................] - ETA: 1s - loss: 0.1060 - accuracy: 0.9667 113/938 [==>...........................] - ETA: 1s - loss: 0.1020 - accuracy: 0.9678 136/938 [===>..........................] - ETA: 1s - loss: 0.1006 - accuracy: 0.9685 160/938 [====>.........................] - ETA: 1s - loss: 0.1018 - accuracy: 0.9688 183/938 [====>.........................] - ETA: 1s - loss: 0.1007 - accuracy: 0.9690 206/938 [=====>........................] - ETA: 1s - loss: 0.0991 - accuracy: 0.9695 229/938 [======>.......................] - ETA: 1s - loss: 0.0989 - accuracy: 0.9694 252/938 [=======>......................] - ETA: 1s - loss: 0.1015 - accuracy: 0.9690 274/938 [=======>......................] - ETA: 1s - loss: 0.1010 - accuracy: 0.9691 297/938 [========>.....................] - ETA: 1s - loss: 0.1013 - accuracy: 0.9690 320/938 [=========>....................] - ETA: 1s - loss: 0.1004 - accuracy: 0.9695 343/938 [=========>....................] - ETA: 1s - loss: 0.1001 - accuracy: 0.9695 366/938 [==========>...................] - ETA: 1s - loss: 0.1010 - accuracy: 0.9690 389/938 [===========>..................] - ETA: 1s - loss: 0.1006 - accuracy: 0.9694 411/938 [============>.................] - ETA: 1s - loss: 0.1002 - accuracy: 0.9697 434/938 [============>.................] - ETA: 1s - loss: 0.1000 - accuracy: 0.9696 457/938 [=============>................] - ETA: 1s - loss: 0.1001 - accuracy: 0.9697 480/938 [==============>...............] - ETA: 1s - loss: 0.1003 - accuracy: 0.9698 502/938 [===============>..............] - ETA: 0s - loss: 0.1006 - accuracy: 0.9695 525/938 [===============>..............] - ETA: 0s - loss: 0.1007 - accuracy: 0.9695 548/938 [================>.............] - ETA: 0s - loss: 0.1003 - accuracy: 0.9697 572/938 [=================>............] - ETA: 0s - loss: 0.1004 - accuracy: 0.9695 596/938 [==================>...........] - ETA: 0s - loss: 0.1015 - accuracy: 0.9691 619/938 [==================>...........] - ETA: 0s - loss: 0.1013 - accuracy: 0.9692 642/938 [===================>..........] - ETA: 0s - loss: 0.1015 - accuracy: 0.9692 666/938 [====================>.........] - ETA: 0s - loss: 0.1014 - accuracy: 0.9692 689/938 [=====================>........] - ETA: 0s - loss: 0.1021 - accuracy: 0.9690 711/938 [=====================>........] - ETA: 0s - loss: 0.1020 - accuracy: 0.9691 735/938 [======================>.......] - ETA: 0s - loss: 0.1016 - accuracy: 0.9692 759/938 [=======================>......] - ETA: 0s - loss: 0.1022 - accuracy: 0.9689 782/938 [========================>.....] - ETA: 0s - loss: 0.1023 - accuracy: 0.9687 805/938 [========================>.....] - ETA: 0s - loss: 0.1032 - accuracy: 0.9684 827/938 [=========================>....] - ETA: 0s - loss: 0.1031 - accuracy: 0.9686 850/938 [==========================>...] - ETA: 0s - loss: 0.1030 - accuracy: 0.9687 872/938 [==========================>...] - ETA: 0s - loss: 0.1028 - accuracy: 0.9686 895/938 [===========================>..] - ETA: 0s - loss: 0.1023 - accuracy: 0.9688 917/938 [============================>.] - ETA: 0s - loss: 0.1024 - accuracy: 0.9688 938/938 [==============================] - 2s 2ms/step - loss: 0.1023 - accuracy: 0.9689 - val_loss: 0.1100 - val_accuracy: 0.9665 Epoch 10/10 1/938 [..............................] - ETA: 2s - loss: 0.0823 - accuracy: 0.9688 24/938 [..............................] - ETA: 1s - loss: 0.0934 - accuracy: 0.9740 47/938 [>.............................] - ETA: 1s - loss: 0.0891 - accuracy: 0.9757 70/938 [=>............................] - ETA: 1s - loss: 0.0892 - accuracy: 0.9746 93/938 [=>............................] - ETA: 1s - loss: 0.0885 - accuracy: 0.9748 115/938 [==>...........................] - ETA: 1s - loss: 0.0886 - accuracy: 0.9746 137/938 [===>..........................] - ETA: 1s - loss: 0.0864 - accuracy: 0.9758 161/938 [====>.........................] - ETA: 1s - loss: 0.0867 - accuracy: 0.9751 184/938 [====>.........................] - ETA: 1s - loss: 0.0878 - accuracy: 0.9743 208/938 [=====>........................] - ETA: 1s - loss: 0.0901 - accuracy: 0.9736 231/938 [======>.......................] - ETA: 1s - loss: 0.0905 - accuracy: 0.9733 254/938 [=======>......................] - ETA: 1s - loss: 0.0918 - accuracy: 0.9732 277/938 [=======>......................] - ETA: 1s - loss: 0.0920 - accuracy: 0.9733 301/938 [========>.....................] - ETA: 1s - loss: 0.0930 - accuracy: 0.9729 324/938 [=========>....................] - ETA: 1s - loss: 0.0941 - accuracy: 0.9725 348/938 [==========>...................] - ETA: 1s - loss: 0.0950 - accuracy: 0.9720 370/938 [==========>...................] - ETA: 1s - loss: 0.0951 - accuracy: 0.9718 391/938 [===========>..................] - ETA: 1s - loss: 0.0961 - accuracy: 0.9714 413/938 [============>.................] - ETA: 1s - loss: 0.0963 - accuracy: 0.9712 435/938 [============>.................] - ETA: 1s - loss: 0.0973 - accuracy: 0.9708 457/938 [=============>................] - ETA: 1s - loss: 0.0974 - accuracy: 0.9708 480/938 [==============>...............] - ETA: 1s - loss: 0.0986 - accuracy: 0.9703 503/938 [===============>..............] - ETA: 0s - loss: 0.0990 - accuracy: 0.9704 525/938 [===============>..............] - ETA: 0s - loss: 0.0995 - accuracy: 0.9703 548/938 [================>.............] - ETA: 0s - loss: 0.1001 - accuracy: 0.9699 569/938 [=================>............] - ETA: 0s - loss: 0.0998 - accuracy: 0.9700 591/938 [=================>............] - ETA: 0s - loss: 0.1003 - accuracy: 0.9698 613/938 [==================>...........] - ETA: 0s - loss: 0.1009 - accuracy: 0.9698 636/938 [===================>..........] - ETA: 0s - loss: 0.1010 - accuracy: 0.9697 658/938 [====================>.........] - ETA: 0s - loss: 0.1015 - accuracy: 0.9696 679/938 [====================>.........] - ETA: 0s - loss: 0.1015 - accuracy: 0.9696 701/938 [=====================>........] - ETA: 0s - loss: 0.1013 - accuracy: 0.9696 723/938 [======================>.......] - ETA: 0s - loss: 0.1010 - accuracy: 0.9697 745/938 [======================>.......] - ETA: 0s - loss: 0.1002 - accuracy: 0.9699 767/938 [=======================>......] - ETA: 0s - loss: 0.1009 - accuracy: 0.9697 789/938 [========================>.....] - ETA: 0s - loss: 0.1010 - accuracy: 0.9697 812/938 [========================>.....] - ETA: 0s - loss: 0.1014 - accuracy: 0.9696 834/938 [=========================>....] - ETA: 0s - loss: 0.1010 - accuracy: 0.9698 857/938 [==========================>...] - ETA: 0s - loss: 0.1007 - accuracy: 0.9699 879/938 [===========================>..] - ETA: 0s - loss: 0.1004 - accuracy: 0.9701 901/938 [===========================>..] - ETA: 0s - loss: 0.1004 - accuracy: 0.9699 922/938 [============================>.] - ETA: 0s - loss: 0.1000 - accuracy: 0.9701 938/938 [==============================] - 2s 3ms/step - loss: 0.1000 - accuracy: 0.9701 - val_loss: 0.1202 - val_accuracy: 0.9644
训练和评估：使用 SGD 作为优化器
model_2 = tf.keras.Sequential([
tf.keras.layers.Dense(64, input_shape=(784,), activation='relu', name='dense_1'),
tf.keras.layers.Dense(64, activation='relu', name='dense_2'),
tf.keras.layers.Dense(10, activation='softmax', name='predictions'),
])
SGD_frobenius_norm_of_weight = []
SGD_get_weight_norm = tf.keras.callbacks.LambdaCallback(
on_epoch_end=lambda batch, logs: SGD_frobenius_norm_of_weight.append(
frobenius_norm(model_2.trainable_weights).numpy()))
# Compile the model
model_2.compile(
optimizer=tf.keras.optimizers.SGD(0.01), # Utilize SGD optimizer
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
metrics=['accuracy'])
history_sgd = model_2.fit(
x_train,
y_train,
batch_size=batch_size,
validation_data=(x_test, y_test),
epochs=epochs,
callbacks=[SGD_get_weight_norm])
Epoch 1/10 1/938 [..............................] - ETA: 3:21 - loss: 2.3501 - accuracy: 0.0625 28/938 [..............................] - ETA: 1s - loss: 2.2752 - accuracy: 0.1445 57/938 [>.............................] - ETA: 1s - loss: 2.2219 - accuracy: 0.1946 85/938 [=>............................] - ETA: 1s - loss: 2.1664 - accuracy: 0.2535 114/938 [==>...........................] - ETA: 1s - loss: 2.1093 - accuracy: 0.3099 143/938 [===>..........................] - ETA: 1s - loss: 2.0465 - accuracy: 0.3639 171/938 [====>.........................] - ETA: 1s - loss: 1.9873 - accuracy: 0.4045 198/938 [=====>........................] - ETA: 1s - loss: 1.9253 - accuracy: 0.4414 225/938 [======>.......................] - ETA: 1s - loss: 1.8606 - accuracy: 0.4740 253/938 [=======>......................] - ETA: 1s - loss: 1.7961 - accuracy: 0.5022 282/938 [========>.....................] - ETA: 1s - loss: 1.7321 - accuracy: 0.5271 310/938 [========>.....................] - ETA: 1s - loss: 1.6734 - accuracy: 0.5468 338/938 [=========>....................] - ETA: 1s - loss: 1.6159 - accuracy: 0.5660 367/938 [==========>...................] - ETA: 1s - loss: 1.5604 - accuracy: 0.5827 396/938 [===========>..................] - ETA: 0s - loss: 1.5073 - accuracy: 0.5986 424/938 [============>.................] - ETA: 0s - loss: 1.4615 - accuracy: 0.6118 452/938 [=============>................] - ETA: 0s - loss: 1.4165 - accuracy: 0.6248 481/938 [==============>...............] - ETA: 0s - loss: 1.3757 - accuracy: 0.6361 509/938 [===============>..............] - ETA: 0s - loss: 1.3384 - accuracy: 0.6465 537/938 [================>.............] - ETA: 0s - loss: 1.3032 - accuracy: 0.6563 565/938 [=================>............] - ETA: 0s - loss: 1.2697 - accuracy: 0.6657 593/938 [=================>............] - ETA: 0s - loss: 1.2386 - accuracy: 0.6737 622/938 [==================>...........] - ETA: 0s - loss: 1.2095 - accuracy: 0.6815 650/938 [===================>..........] - ETA: 0s - loss: 1.1825 - accuracy: 0.6888 679/938 [====================>.........] - ETA: 0s - loss: 1.1543 - accuracy: 0.6964 708/938 [=====================>........] - ETA: 0s - loss: 1.1280 - accuracy: 0.7032 737/938 [======================>.......] - ETA: 0s - loss: 1.1046 - accuracy: 0.7096 766/938 [=======================>......] - ETA: 0s - loss: 1.0818 - accuracy: 0.7156 795/938 [========================>.....] - ETA: 0s - loss: 1.0602 - accuracy: 0.7212 824/938 [=========================>....] - ETA: 0s - loss: 1.0408 - accuracy: 0.7261 853/938 [==========================>...] - ETA: 0s - loss: 1.0221 - accuracy: 0.7312 880/938 [===========================>..] - ETA: 0s - loss: 1.0054 - accuracy: 0.7352 908/938 [============================>.] - ETA: 0s - loss: 0.9898 - accuracy: 0.7394 936/938 [============================>.] - ETA: 0s - loss: 0.9741 - accuracy: 0.7433 938/938 [==============================] - 2s 2ms/step - loss: 0.9734 - accuracy: 0.7435 - val_loss: 0.4342 - val_accuracy: 0.8835 Epoch 2/10 1/938 [..............................] - ETA: 1s - loss: 0.5865 - accuracy: 0.8125 30/938 [..............................] - ETA: 1s - loss: 0.4576 - accuracy: 0.8719 59/938 [>.............................] - ETA: 1s - loss: 0.4374 - accuracy: 0.8798 88/938 [=>............................] - ETA: 1s - loss: 0.4375 - accuracy: 0.8821 116/938 [==>...........................] - ETA: 1s - loss: 0.4348 - accuracy: 0.8808 144/938 [===>..........................] - ETA: 1s - loss: 0.4332 - accuracy: 0.8816 173/938 [====>.........................] - ETA: 1s - loss: 0.4297 - accuracy: 0.8820 199/938 [=====>........................] - ETA: 1s - loss: 0.4295 - accuracy: 0.8819 225/938 [======>.......................] - ETA: 1s - loss: 0.4289 - accuracy: 0.8822 251/938 [=======>......................] - ETA: 1s - loss: 0.4270 - accuracy: 0.8831 277/938 [=======>......................] - ETA: 1s - loss: 0.4302 - accuracy: 0.8824 303/938 [========>.....................] - ETA: 1s - loss: 0.4262 - accuracy: 0.8841 330/938 [=========>....................] - ETA: 1s - loss: 0.4273 - accuracy: 0.8837 357/938 [==========>...................] - ETA: 1s - loss: 0.4249 - accuracy: 0.8842 384/938 [===========>..................] - ETA: 1s - loss: 0.4222 - accuracy: 0.8845 411/938 [============>.................] - ETA: 0s - loss: 0.4213 - accuracy: 0.8847 439/938 [=============>................] - ETA: 0s - loss: 0.4207 - accuracy: 0.8847 468/938 [=============>................] - ETA: 0s - loss: 0.4189 - accuracy: 0.8848 496/938 [==============>...............] - ETA: 0s - loss: 0.4170 - accuracy: 0.8851 525/938 [===============>..............] - ETA: 0s - loss: 0.4150 - accuracy: 0.8856 554/938 [================>.............] - ETA: 0s - loss: 0.4135 - accuracy: 0.8860 583/938 [=================>............] - ETA: 0s - loss: 0.4102 - accuracy: 0.8866 609/938 [==================>...........] - ETA: 0s - loss: 0.4077 - accuracy: 0.8868 637/938 [===================>..........] - ETA: 0s - loss: 0.4053 - accuracy: 0.8875 665/938 [====================>.........] - ETA: 0s - loss: 0.4042 - accuracy: 0.8876 695/938 [=====================>........] - ETA: 0s - loss: 0.4029 - accuracy: 0.8880 723/938 [======================>.......] - ETA: 0s - loss: 0.4010 - accuracy: 0.8888 751/938 [=======================>......] - ETA: 0s - loss: 0.3982 - accuracy: 0.8895 780/938 [=======================>......] - ETA: 0s - loss: 0.3966 - accuracy: 0.8896 810/938 [========================>.....] - ETA: 0s - loss: 0.3972 - accuracy: 0.8896 837/938 [=========================>....] - ETA: 0s - loss: 0.3961 - accuracy: 0.8899 865/938 [==========================>...] - ETA: 0s - loss: 0.3940 - accuracy: 0.8904 894/938 [===========================>..] - ETA: 0s - loss: 0.3913 - accuracy: 0.8910 923/938 [============================>.] - ETA: 0s - loss: 0.3888 - accuracy: 0.8917 938/938 [==============================] - 2s 2ms/step - loss: 0.3880 - accuracy: 0.8919 - val_loss: 0.3311 - val_accuracy: 0.9070 Epoch 3/10 1/938 [..............................] - ETA: 2s - loss: 0.1631 - accuracy: 0.9688 29/938 [..............................] - ETA: 1s - loss: 0.3612 - accuracy: 0.8874 57/938 [>.............................] - ETA: 1s - loss: 0.3552 - accuracy: 0.8961 86/938 [=>............................] - ETA: 1s - loss: 0.3497 - accuracy: 0.8990 115/938 [==>...........................] - ETA: 1s - loss: 0.3431 - accuracy: 0.9022 143/938 [===>..........................] - ETA: 1s - loss: 0.3401 - accuracy: 0.9017 172/938 [====>.........................] - ETA: 1s - loss: 0.3348 - accuracy: 0.9034 200/938 [=====>........................] - ETA: 1s - loss: 0.3309 - accuracy: 0.9051 228/938 [======>.......................] - ETA: 1s - loss: 0.3312 - accuracy: 0.9052 257/938 [=======>......................] - ETA: 1s - loss: 0.3308 - accuracy: 0.9057 286/938 [========>.....................] - ETA: 1s - loss: 0.3306 - accuracy: 0.9056 313/938 [=========>....................] - ETA: 1s - loss: 0.3331 - accuracy: 0.9049 340/938 [=========>....................] - ETA: 1s - loss: 0.3320 - accuracy: 0.9055 368/938 [==========>...................] - ETA: 1s - loss: 0.3340 - accuracy: 0.9054 396/938 [===========>..................] - ETA: 0s - loss: 0.3340 - accuracy: 0.9056 424/938 [============>.................] - ETA: 0s - loss: 0.3325 - accuracy: 0.9060 452/938 [=============>................] - ETA: 0s - loss: 0.3316 - accuracy: 0.9063 479/938 [==============>...............] - ETA: 0s - loss: 0.3297 - accuracy: 0.9073 507/938 [===============>..............] - ETA: 0s - loss: 0.3273 - accuracy: 0.9079 536/938 [================>.............] - ETA: 0s - loss: 0.3281 - accuracy: 0.9076 565/938 [=================>............] - ETA: 0s - loss: 0.3266 - accuracy: 0.9081 594/938 [=================>............] - ETA: 0s - loss: 0.3257 - accuracy: 0.9083 623/938 [==================>...........] - ETA: 0s - loss: 0.3274 - accuracy: 0.9079 650/938 [===================>..........] - ETA: 0s - loss: 0.3271 - accuracy: 0.9080 679/938 [====================>.........] - ETA: 0s - loss: 0.3265 - accuracy: 0.9078 707/938 [=====================>........] - ETA: 0s - loss: 0.3261 - accuracy: 0.9080 736/938 [======================>.......] - ETA: 0s - loss: 0.3269 - accuracy: 0.9077 764/938 [=======================>......] - ETA: 0s - loss: 0.3267 - accuracy: 0.9078 793/938 [========================>.....] - ETA: 0s - loss: 0.3264 - accuracy: 0.9078 821/938 [=========================>....] - ETA: 0s - loss: 0.3259 - accuracy: 0.9079 850/938 [==========================>...] - ETA: 0s - loss: 0.3251 - accuracy: 0.9079 877/938 [===========================>..] - ETA: 0s - loss: 0.3238 - accuracy: 0.9085 906/938 [===========================>..] - ETA: 0s - loss: 0.3239 - accuracy: 0.9085 934/938 [============================>.] - ETA: 0s - loss: 0.3227 - accuracy: 0.9087 938/938 [==============================] - 2s 2ms/step - loss: 0.3228 - accuracy: 0.9086 - val_loss: 0.2885 - val_accuracy: 0.9182 Epoch 4/10 1/938 [..............................] - ETA: 1s - loss: 0.2661 - accuracy: 0.9219 30/938 [..............................] - ETA: 1s - loss: 0.2969 - accuracy: 0.9182 59/938 [>.............................] - ETA: 1s - loss: 0.3040 - accuracy: 0.9139 88/938 [=>............................] - ETA: 1s - loss: 0.3040 - accuracy: 0.9148 117/938 [==>...........................] - ETA: 1s - loss: 0.3054 - accuracy: 0.9129 146/938 [===>..........................] - ETA: 1s - loss: 0.2976 - accuracy: 0.9149 173/938 [====>.........................] - ETA: 1s - loss: 0.2977 - accuracy: 0.9141 200/938 [=====>........................] - ETA: 1s - loss: 0.2967 - accuracy: 0.9148 228/938 [======>.......................] - ETA: 1s - loss: 0.2935 - accuracy: 0.9155 256/938 [=======>......................] - ETA: 1s - loss: 0.2950 - accuracy: 0.9152 285/938 [========>.....................] - ETA: 1s - loss: 0.2946 - accuracy: 0.9150 314/938 [=========>....................] - ETA: 1s - loss: 0.2953 - accuracy: 0.9146 342/938 [=========>....................] - ETA: 1s - loss: 0.2963 - accuracy: 0.9145 370/938 [==========>...................] - ETA: 1s - loss: 0.2975 - accuracy: 0.9139 399/938 [===========>..................] - ETA: 0s - loss: 0.2967 - accuracy: 0.9137 428/938 [============>.................] - ETA: 0s - loss: 0.2969 - accuracy: 0.9133 456/938 [=============>................] - ETA: 0s - loss: 0.2969 - accuracy: 0.9138 486/938 [==============>...............] - ETA: 0s - loss: 0.2961 - accuracy: 0.9138 515/938 [===============>..............] - ETA: 0s - loss: 0.2963 - accuracy: 0.9139 541/938 [================>.............] - ETA: 0s - loss: 0.2972 - accuracy: 0.9137 566/938 [=================>............] - ETA: 0s - loss: 0.2974 - accuracy: 0.9137 593/938 [=================>............] - ETA: 0s - loss: 0.2952 - accuracy: 0.9148 622/938 [==================>...........] - ETA: 0s - loss: 0.2950 - accuracy: 0.9150 650/938 [===================>..........] - ETA: 0s - loss: 0.2943 - accuracy: 0.9152 677/938 [====================>.........] - ETA: 0s - loss: 0.2939 - accuracy: 0.9153 706/938 [=====================>........] - ETA: 0s - loss: 0.2935 - accuracy: 0.9150 733/938 [======================>.......] - ETA: 0s - loss: 0.2931 - accuracy: 0.9151 761/938 [=======================>......] - ETA: 0s - loss: 0.2912 - accuracy: 0.9155 788/938 [========================>.....] - ETA: 0s - loss: 0.2904 - accuracy: 0.9159 815/938 [=========================>....] - ETA: 0s - loss: 0.2898 - accuracy: 0.9163 842/938 [=========================>....] - ETA: 0s - loss: 0.2902 - accuracy: 0.9161 870/938 [==========================>...] - ETA: 0s - loss: 0.2889 - accuracy: 0.9164 899/938 [===========================>..] - ETA: 0s - loss: 0.2892 - accuracy: 0.9166 927/938 [============================>.] - ETA: 0s - loss: 0.2890 - accuracy: 0.9166 938/938 [==============================] - 2s 2ms/step - loss: 0.2890 - accuracy: 0.9166 - val_loss: 0.2661 - val_accuracy: 0.9233 Epoch 5/10 1/938 [..............................] - ETA: 2s - loss: 0.2592 - accuracy: 0.9062 28/938 [..............................] - ETA: 1s - loss: 0.2461 - accuracy: 0.9314 55/938 [>.............................] - ETA: 1s - loss: 0.2638 - accuracy: 0.9233 82/938 [=>............................] - ETA: 1s - loss: 0.2591 - accuracy: 0.9244 110/938 [==>...........................] - ETA: 1s - loss: 0.2744 - accuracy: 0.9203 137/938 [===>..........................] - ETA: 1s - loss: 0.2756 - accuracy: 0.9205 164/938 [====>.........................] - ETA: 1s - loss: 0.2747 - accuracy: 0.9199 192/938 [=====>........................] - ETA: 1s - loss: 0.2741 - accuracy: 0.9208 220/938 [======>.......................] - ETA: 1s - loss: 0.2727 - accuracy: 0.9210 248/938 [======>.......................] - ETA: 1s - loss: 0.2698 - accuracy: 0.9214 276/938 [=======>......................] - ETA: 1s - loss: 0.2707 - accuracy: 0.9218 305/938 [========>.....................] - ETA: 1s - loss: 0.2712 - accuracy: 0.9218 334/938 [=========>....................] - ETA: 1s - loss: 0.2755 - accuracy: 0.9207 362/938 [==========>...................] - ETA: 1s - loss: 0.2758 - accuracy: 0.9207 391/938 [===========>..................] - ETA: 0s - loss: 0.2746 - accuracy: 0.9208 420/938 [============>.................] - ETA: 0s - loss: 0.2739 - accuracy: 0.9213 448/938 [=============>................] - ETA: 0s - loss: 0.2746 - accuracy: 0.9207 476/938 [==============>...............] - ETA: 0s - loss: 0.2769 - accuracy: 0.9201 504/938 [===============>..............] - ETA: 0s - loss: 0.2754 - accuracy: 0.9208 531/938 [===============>..............] - ETA: 0s - loss: 0.2748 - accuracy: 0.9207 559/938 [================>.............] - ETA: 0s - loss: 0.2751 - accuracy: 0.9206 588/938 [=================>............] - ETA: 0s - loss: 0.2738 - accuracy: 0.9208 616/938 [==================>...........] - ETA: 0s - loss: 0.2711 - accuracy: 0.9217 643/938 [===================>..........] - ETA: 0s - loss: 0.2704 - accuracy: 0.9220 671/938 [====================>.........] - ETA: 0s - loss: 0.2703 - accuracy: 0.9221 699/938 [=====================>........] - ETA: 0s - loss: 0.2694 - accuracy: 0.9224 729/938 [======================>.......] - ETA: 0s - loss: 0.2690 - accuracy: 0.9226 758/938 [=======================>......] - ETA: 0s - loss: 0.2693 - accuracy: 0.9227 786/938 [========================>.....] - ETA: 0s - loss: 0.2685 - accuracy: 0.9228 815/938 [=========================>....] - ETA: 0s - loss: 0.2678 - accuracy: 0.9230 843/938 [=========================>....] - ETA: 0s - loss: 0.2680 - accuracy: 0.9229 870/938 [==========================>...] - ETA: 0s - loss: 0.2675 - accuracy: 0.9233 897/938 [===========================>..] - ETA: 0s - loss: 0.2664 - accuracy: 0.9236 924/938 [============================>.] - ETA: 0s - loss: 0.2656 - accuracy: 0.9238 938/938 [==============================] - 2s 2ms/step - loss: 0.2650 - accuracy: 0.9238 - val_loss: 0.2466 - val_accuracy: 0.9301 Epoch 6/10 1/938 [..............................] - ETA: 1s - loss: 0.1986 - accuracy: 0.9531 30/938 [..............................] - ETA: 1s - loss: 0.2620 - accuracy: 0.9240 59/938 [>.............................] - ETA: 1s - loss: 0.2649 - accuracy: 0.9213 88/938 [=>............................] - ETA: 1s - loss: 0.2670 - accuracy: 0.9215 118/938 [==>...........................] - ETA: 1s - loss: 0.2602 - accuracy: 0.9236 147/938 [===>..........................] - ETA: 1s - loss: 0.2611 - accuracy: 0.9233 175/938 [====>.........................] - ETA: 1s - loss: 0.2594 - accuracy: 0.9235 205/938 [=====>........................] - ETA: 1s - loss: 0.2566 - accuracy: 0.9248 234/938 [======>.......................] - ETA: 1s - loss: 0.2559 - accuracy: 0.9249 264/938 [=======>......................] - ETA: 1s - loss: 0.2544 - accuracy: 0.9261 292/938 [========>.....................] - ETA: 1s - loss: 0.2528 - accuracy: 0.9266 320/938 [=========>....................] - ETA: 1s - loss: 0.2537 - accuracy: 0.9266 346/938 [==========>...................] - ETA: 1s - loss: 0.2526 - accuracy: 0.9267 373/938 [==========>...................] - ETA: 1s - loss: 0.2521 - accuracy: 0.9273 401/938 [===========>..................] - ETA: 0s - loss: 0.2512 - accuracy: 0.9278 429/938 [============>.................] - ETA: 0s - loss: 0.2506 - accuracy: 0.9281 456/938 [=============>................] - ETA: 0s - loss: 0.2502 - accuracy: 0.9285 483/938 [==============>...............] - ETA: 0s - loss: 0.2508 - accuracy: 0.9283 511/938 [===============>..............] - ETA: 0s - loss: 0.2494 - accuracy: 0.9288 538/938 [================>.............] - ETA: 0s - loss: 0.2477 - accuracy: 0.9291 566/938 [=================>............] - ETA: 0s - loss: 0.2477 - accuracy: 0.9293 594/938 [=================>............] - ETA: 0s - loss: 0.2472 - accuracy: 0.9292 621/938 [==================>...........] - ETA: 0s - loss: 0.2468 - accuracy: 0.9294 648/938 [===================>..........] - ETA: 0s - loss: 0.2469 - accuracy: 0.9294 676/938 [====================>.........] - ETA: 0s - loss: 0.2468 - accuracy: 0.9295 704/938 [=====================>........] - ETA: 0s - loss: 0.2470 - accuracy: 0.9292 732/938 [======================>.......] - ETA: 0s - loss: 0.2465 - accuracy: 0.9295 761/938 [=======================>......] - ETA: 0s - loss: 0.2461 - accuracy: 0.9296 789/938 [========================>.....] - ETA: 0s - loss: 0.2452 - accuracy: 0.9299 818/938 [=========================>....] - ETA: 0s - loss: 0.2441 - accuracy: 0.9302 846/938 [==========================>...] - ETA: 0s - loss: 0.2446 - accuracy: 0.9299 874/938 [==========================>...] - ETA: 0s - loss: 0.2455 - accuracy: 0.9296 901/938 [===========================>..] - ETA: 0s - loss: 0.2457 - accuracy: 0.9296 928/938 [============================>.] - ETA: 0s - loss: 0.2457 - accuracy: 0.9295 938/938 [==============================] - 2s 2ms/step - loss: 0.2457 - accuracy: 0.9295 - val_loss: 0.2293 - val_accuracy: 0.9350 Epoch 7/10 1/938 [..............................] - ETA: 2s - loss: 0.3150 - accuracy: 0.9375 29/938 [..............................] - ETA: 1s - loss: 0.2102 - accuracy: 0.9445 56/938 [>.............................] - ETA: 1s - loss: 0.2241 - accuracy: 0.9375 83/938 [=>............................] - ETA: 1s - loss: 0.2235 - accuracy: 0.9381 111/938 [==>...........................] - ETA: 1s - loss: 0.2240 - accuracy: 0.9369 139/938 [===>..........................] - ETA: 1s - loss: 0.2241 - accuracy: 0.9354 167/938 [====>.........................] - ETA: 1s - loss: 0.2228 - accuracy: 0.9353 194/938 [=====>........................] - ETA: 1s - loss: 0.2259 - accuracy: 0.9338 221/938 [======>.......................] - ETA: 1s - loss: 0.2274 - accuracy: 0.9329 250/938 [======>.......................] - ETA: 1s - loss: 0.2287 - accuracy: 0.9325 279/938 [=======>......................] - ETA: 1s - loss: 0.2310 - accuracy: 0.9326 306/938 [========>.....................] - ETA: 1s - loss: 0.2309 - accuracy: 0.9325 333/938 [=========>....................] - ETA: 1s - loss: 0.2316 - accuracy: 0.9328 360/938 [==========>...................] - ETA: 1s - loss: 0.2322 - accuracy: 0.9331 389/938 [===========>..................] - ETA: 1s - loss: 0.2312 - accuracy: 0.9336 416/938 [============>.................] - ETA: 0s - loss: 0.2299 - accuracy: 0.9341 444/938 [=============>................] - ETA: 0s - loss: 0.2292 - accuracy: 0.9344 473/938 [==============>...............] - ETA: 0s - loss: 0.2289 - accuracy: 0.9340 502/938 [===============>..............] - ETA: 0s - loss: 0.2284 - accuracy: 0.9342 530/938 [===============>..............] - ETA: 0s - loss: 0.2278 - accuracy: 0.9343 558/938 [================>.............] - ETA: 0s - loss: 0.2274 - accuracy: 0.9342 587/938 [=================>............] - ETA: 0s - loss: 0.2265 - accuracy: 0.9345 615/938 [==================>...........] - ETA: 0s - loss: 0.2266 - accuracy: 0.9347 642/938 [===================>..........] - ETA: 0s - loss: 0.2266 - accuracy: 0.9347 670/938 [====================>.........] - ETA: 0s - loss: 0.2248 - accuracy: 0.9351 698/938 [=====================>........] - ETA: 0s - loss: 0.2263 - accuracy: 0.9344 726/938 [======================>.......] - ETA: 0s - loss: 0.2270 - accuracy: 0.9343 752/938 [=======================>......] - ETA: 0s - loss: 0.2277 - accuracy: 0.9340 779/938 [=======================>......] - ETA: 0s - loss: 0.2271 - accuracy: 0.9341 806/938 [========================>.....] - ETA: 0s - loss: 0.2263 - accuracy: 0.9342 834/938 [=========================>....] - ETA: 0s - loss: 0.2272 - accuracy: 0.9341 862/938 [==========================>...] - ETA: 0s - loss: 0.2276 - accuracy: 0.9342 890/938 [===========================>..] - ETA: 0s - loss: 0.2289 - accuracy: 0.9337 918/938 [============================>.] - ETA: 0s - loss: 0.2296 - accuracy: 0.9336 938/938 [==============================] - 2s 2ms/step - loss: 0.2295 - accuracy: 0.9334 - val_loss: 0.2153 - val_accuracy: 0.9385 Epoch 8/10 1/938 [..............................] - ETA: 1s - loss: 0.2166 - accuracy: 0.9531 30/938 [..............................] - ETA: 1s - loss: 0.2303 - accuracy: 0.9328 57/938 [>.............................] - ETA: 1s - loss: 0.2228 - accuracy: 0.9345 86/938 [=>............................] - ETA: 1s - loss: 0.2155 - accuracy: 0.9379 114/938 [==>...........................] - ETA: 1s - loss: 0.2200 - accuracy: 0.9361 142/938 [===>..........................] - ETA: 1s - loss: 0.2154 - accuracy: 0.9377 170/938 [====>.........................] - ETA: 1s - loss: 0.2198 - accuracy: 0.9377 198/938 [=====>........................] - ETA: 1s - loss: 0.2230 - accuracy: 0.9361 226/938 [======>.......................] - ETA: 1s - loss: 0.2245 - accuracy: 0.9361 256/938 [=======>......................] - ETA: 1s - loss: 0.2240 - accuracy: 0.9360 284/938 [========>.....................] - ETA: 1s - loss: 0.2222 - accuracy: 0.9365 313/938 [=========>....................] - ETA: 1s - loss: 0.2206 - accuracy: 0.9368 340/938 [=========>....................] - ETA: 1s - loss: 0.2184 - accuracy: 0.9371 367/938 [==========>...................] - ETA: 1s - loss: 0.2185 - accuracy: 0.9373 394/938 [===========>..................] - ETA: 0s - loss: 0.2183 - accuracy: 0.9375 420/938 [============>.................] - ETA: 0s - loss: 0.2182 - accuracy: 0.9378 448/938 [=============>................] - ETA: 0s - loss: 0.2182 - accuracy: 0.9375 476/938 [==============>...............] - ETA: 0s - loss: 0.2171 - accuracy: 0.9381 504/938 [===============>..............] - ETA: 0s - loss: 0.2182 - accuracy: 0.9378 533/938 [================>.............] - ETA: 0s - loss: 0.2179 - accuracy: 0.9376 561/938 [================>.............] - ETA: 0s - loss: 0.2172 - accuracy: 0.9378 590/938 [=================>............] - ETA: 0s - loss: 0.2163 - accuracy: 0.9377 618/938 [==================>...........] - ETA: 0s - loss: 0.2152 - accuracy: 0.9377 647/938 [===================>..........] - ETA: 0s - loss: 0.2150 - accuracy: 0.9379 675/938 [====================>.........] - ETA: 0s - loss: 0.2139 - accuracy: 0.9384 703/938 [=====================>........] - ETA: 0s - loss: 0.2143 - accuracy: 0.9381 731/938 [======================>.......] - ETA: 0s - loss: 0.2163 - accuracy: 0.9374 760/938 [=======================>......] - ETA: 0s - loss: 0.2153 - accuracy: 0.9376 788/938 [========================>.....] - ETA: 0s - loss: 0.2167 - accuracy: 0.9373 817/938 [=========================>....] - ETA: 0s - loss: 0.2163 - accuracy: 0.9377 845/938 [==========================>...] - ETA: 0s - loss: 0.2158 - accuracy: 0.9381 875/938 [==========================>...] - ETA: 0s - loss: 0.2155 - accuracy: 0.9378 904/938 [===========================>..] - ETA: 0s - loss: 0.2155 - accuracy: 0.9377 932/938 [============================>.] - ETA: 0s - loss: 0.2150 - accuracy: 0.9378 938/938 [==============================] - 2s 2ms/step - loss: 0.2153 - accuracy: 0.9377 - val_loss: 0.2031 - val_accuracy: 0.9405 Epoch 9/10 1/938 [..............................] - ETA: 2s - loss: 0.2150 - accuracy: 0.9062 30/938 [..............................] - ETA: 1s - loss: 0.1848 - accuracy: 0.9469 58/938 [>.............................] - ETA: 1s - loss: 0.2137 - accuracy: 0.9375 86/938 [=>............................] - ETA: 1s - loss: 0.2173 - accuracy: 0.9370 114/938 [==>...........................] - ETA: 1s - loss: 0.2168 - accuracy: 0.9359 142/938 [===>..........................] - ETA: 1s - loss: 0.2155 - accuracy: 0.9368 170/938 [====>.........................] - ETA: 1s - loss: 0.2128 - accuracy: 0.9379 199/938 [=====>........................] - ETA: 1s - loss: 0.2117 - accuracy: 0.9388 227/938 [======>.......................] - ETA: 1s - loss: 0.2103 - accuracy: 0.9389 255/938 [=======>......................] - ETA: 1s - loss: 0.2126 - accuracy: 0.9385 284/938 [========>.....................] - ETA: 1s - loss: 0.2108 - accuracy: 0.9392 311/938 [========>.....................] - ETA: 1s - loss: 0.2088 - accuracy: 0.9399 340/938 [=========>....................] - ETA: 1s - loss: 0.2119 - accuracy: 0.9391 370/938 [==========>...................] - ETA: 1s - loss: 0.2101 - accuracy: 0.9397 399/938 [===========>..................] - ETA: 0s - loss: 0.2103 - accuracy: 0.9397 429/938 [============>.................] - ETA: 0s - loss: 0.2104 - accuracy: 0.9392 458/938 [=============>................] - ETA: 0s - loss: 0.2091 - accuracy: 0.9395 486/938 [==============>...............] - ETA: 0s - loss: 0.2086 - accuracy: 0.9396 515/938 [===============>..............] - ETA: 0s - loss: 0.2072 - accuracy: 0.9399 545/938 [================>.............] - ETA: 0s - loss: 0.2077 - accuracy: 0.9401 574/938 [=================>............] - ETA: 0s - loss: 0.2080 - accuracy: 0.9400 603/938 [==================>...........] - ETA: 0s - loss: 0.2080 - accuracy: 0.9397 632/938 [===================>..........] - ETA: 0s - loss: 0.2083 - accuracy: 0.9396 661/938 [====================>.........] - ETA: 0s - loss: 0.2076 - accuracy: 0.9394 689/938 [=====================>........] - ETA: 0s - loss: 0.2070 - accuracy: 0.9398 718/938 [=====================>........] - ETA: 0s - loss: 0.2068 - accuracy: 0.9399 747/938 [======================>.......] - ETA: 0s - loss: 0.2057 - accuracy: 0.9401 776/938 [=======================>......] - ETA: 0s - loss: 0.2056 - accuracy: 0.9401 805/938 [========================>.....] - ETA: 0s - loss: 0.2054 - accuracy: 0.9401 834/938 [=========================>....] - ETA: 0s - loss: 0.2049 - accuracy: 0.9403 863/938 [==========================>...] - ETA: 0s - loss: 0.2038 - accuracy: 0.9408 892/938 [===========================>..] - ETA: 0s - loss: 0.2031 - accuracy: 0.9408 922/938 [============================>.] - ETA: 0s - loss: 0.2029 - accuracy: 0.9409 938/938 [==============================] - 2s 2ms/step - loss: 0.2027 - accuracy: 0.9409 - val_loss: 0.1936 - val_accuracy: 0.9436 Epoch 10/10 1/938 [..............................] - ETA: 1s - loss: 0.1608 - accuracy: 0.9531 30/938 [..............................] - ETA: 1s - loss: 0.1973 - accuracy: 0.9401 59/938 [>.............................] - ETA: 1s - loss: 0.1935 - accuracy: 0.9439 87/938 [=>............................] - ETA: 1s - loss: 0.1916 - accuracy: 0.9465 116/938 [==>...........................] - ETA: 1s - loss: 0.1877 - accuracy: 0.9467 145/938 [===>..........................] - ETA: 1s - loss: 0.1899 - accuracy: 0.9457 174/938 [====>.........................] - ETA: 1s - loss: 0.1908 - accuracy: 0.9453 202/938 [=====>........................] - ETA: 1s - loss: 0.1929 - accuracy: 0.9438 230/938 [======>.......................] - ETA: 1s - loss: 0.1927 - accuracy: 0.9439 258/938 [=======>......................] - ETA: 1s - loss: 0.1931 - accuracy: 0.9442 286/938 [========>.....................] - ETA: 1s - loss: 0.1934 - accuracy: 0.9443 315/938 [=========>....................] - ETA: 1s - loss: 0.1932 - accuracy: 0.9444 344/938 [==========>...................] - ETA: 1s - loss: 0.1920 - accuracy: 0.9446 373/938 [==========>...................] - ETA: 1s - loss: 0.1923 - accuracy: 0.9450 401/938 [===========>..................] - ETA: 0s - loss: 0.1929 - accuracy: 0.9446 429/938 [============>.................] - ETA: 0s - loss: 0.1933 - accuracy: 0.9446 457/938 [=============>................] - ETA: 0s - loss: 0.1931 - accuracy: 0.9443 485/938 [==============>...............] - ETA: 0s - loss: 0.1921 - accuracy: 0.9443 513/938 [===============>..............] - ETA: 0s - loss: 0.1941 - accuracy: 0.9441 541/938 [================>.............] - ETA: 0s - loss: 0.1950 - accuracy: 0.9439 570/938 [=================>............] - ETA: 0s - loss: 0.1946 - accuracy: 0.9441 599/938 [==================>...........] - ETA: 0s - loss: 0.1934 - accuracy: 0.9444 627/938 [===================>..........] - ETA: 0s - loss: 0.1936 - accuracy: 0.9442 656/938 [===================>..........] - ETA: 0s - loss: 0.1931 - accuracy: 0.9443 685/938 [====================>.........] - ETA: 0s - loss: 0.1936 - accuracy: 0.9441 715/938 [=====================>........] - ETA: 0s - loss: 0.1940 - accuracy: 0.9436 746/938 [======================>.......] - ETA: 0s - loss: 0.1940 - accuracy: 0.9438 776/938 [=======================>......] - ETA: 0s - loss: 0.1937 - accuracy: 0.9438 805/938 [========================>.....] - ETA: 0s - loss: 0.1926 - accuracy: 0.9442 835/938 [=========================>....] - ETA: 0s - loss: 0.1924 - accuracy: 0.9442 864/938 [==========================>...] - ETA: 0s - loss: 0.1916 - accuracy: 0.9445 893/938 [===========================>..] - ETA: 0s - loss: 0.1918 - accuracy: 0.9444 923/938 [============================>.] - ETA: 0s - loss: 0.1917 - accuracy: 0.9444 938/938 [==============================] - 2s 2ms/step - loss: 0.1918 - accuracy: 0.9443 - val_loss: 0.1853 - val_accuracy: 0.9447
权重的弗罗宾尼斯范数：CG 与 SGD
CG 优化器的当前实现基于弗罗宾尼斯范数，并考虑将弗罗宾尼斯范数作为目标函数中的正则化器。因此，我们将 CG 的正则化效果与尚未采用弗罗宾尼斯范数正则化器的 SGD 优化器进行比较。
plt.plot(
CG_frobenius_norm_of_weight,
color='r',
label='CG_frobenius_norm_of_weights')
plt.plot(
SGD_frobenius_norm_of_weight,
color='b',
label='SGD_frobenius_norm_of_weights')
plt.xlabel('Epoch')
plt.ylabel('Frobenius norm of weights')
plt.legend(loc=1)
<matplotlib.legend.Legend at 0x7efde5ac4c50>
训练和验证准确率：CG 与 SGD
plt.plot(history_cg.history['accuracy'], color='r', label='CG_train')
plt.plot(history_cg.history['val_accuracy'], color='g', label='CG_test')
plt.plot(history_sgd.history['accuracy'], color='pink', label='SGD_train')
plt.plot(history_sgd.history['val_accuracy'], color='b', label='SGD_test')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend(loc=4)
<matplotlib.legend.Legend at 0x7efde514fe10>
