|---|---|---|---|
| [View on TensorFlow.org](https://www.tensorflow.org/text/tutorials/warmstart_embedding_matrix.md.txt) | [Run in Google Colab](https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/warmstart_embedding_matrix.ipynb) | [View on GitHub](https://github.com/tensorflow/text/blob/master/docs/tutorials/warmstart_embedding_matrix.ipynb) | [Download notebook](https://storage.googleapis.com/tensorflow_docs/text/docs/tutorials/warmstart_embedding_matrix.ipynb) |
This tutorial shows how to "warm-start" training using the [`tf.keras.utils.warmstart_embedding_matrix`](https://www.tensorflow.org/api_docs/python/tf/keras/utils/warmstart_embedding_matrix) API for text sentiment classification when changing vocabulary.
You will begin by training a simple Keras model with a base vocabulary, and then, after updating the vocabulary, continue training the model. This is referred to as "warm-start" training, for which you'll need to remap the text-embedding matrix for the new vocabulary.
## Embedding matrix
Embeddings provide a way to use an efficient, dense representation in which similar vocabulary tokens have a similar encoding. They are trainable parameters (weights learned by the model during training, in the same way a model learns weights for a dense layer). It is common to have embeddings that are 8-dimensional for small datasets, and up to 1024-dimensions when working with large datasets. A higher dimensional embedding can capture fine-grained relationships between words, but can take more data to learn.
### Vocabulary
The set of unique words is referred to as the vocabulary. To build a text model you need to choose a fixed vocabulary. Typically you build the vocabulary from the most common words in a dataset. The vocabulary allows us to represent each piece of text by a sequence of ID's that you can lookup in the embedding matrix. Vocabulary allows us to represent each piece of text by the specific words that appear in it.
### Why warm-start an embedding matrix?
A model is trained with a set of embeddings that represents a given vocabulary. If the model needs to be updated or improved you can train to convergence significantly faster by reusing weights from a previous run. Using the embedding matrix from a previous run is more difficult. The problem is that any change to the vocabulary invalidates the word to id mapping.
The `tf.keras.utils.warmstart_embedding_matrix` solves this problem by creating an embedding matrix for a new vocabulary from an embedding matrix from a base vocabulary. Where a word exists in both vocabularies the base embedding vector is copied into the correct location in the new embedding matrix. This allows you to warm-start training after any change in the size or order of the vocabulary.
## Setup
```bash
pip install -q -U tf-keras
```
```python
import io
import numpy as np
import os
import re
import shutil
import string
os.environ["TF_USE_LEGACY_KERAS"] = "1"
import tensorflow as tf
from tensorflow.keras import Model
from tensorflow.keras.layers import Dense, Embedding, GlobalAveragePooling1D
from tensorflow.keras.layers import TextVectorization
```
### Load the dataset
The tutorial uses the [Large Movie Review Dataset](http://ai.stanford.edu/%7Eamaas/data/sentiment/). You will train a sentiment classifier model on this dataset and in the process learn embeddings from scratch. Refer to the [Loading text tutorial](https://www.tensorflow.org/tutorials/load_data/text.md.txt) to learn more.
Download the dataset using Keras file utility and review the directories.
```python
url = "https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
dataset = tf.keras.utils.get_file(
"aclImdb_v1.tar.gz", url, untar=True, cache_dir=".", cache_subdir=""
)
dataset_dir = os.path.join(os.path.dirname(dataset), "aclImdb")
if not os.path.exists(dataset_dir):
dataset_dir = os.path.join(dataset, "aclImdb")
os.listdir(dataset_dir)
```
```
Downloading data from https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
84125825/84125825 [==============================] - 1s 0us/step
['test', 'README', 'train', 'imdb.vocab', 'imdbEr.txt']
```
The `train/` directory has `pos` and `neg` folders with movie reviews labeled as positive and negative respectively. You will use reviews from `pos` and `neg` folders to train a binary classification model.
```python
train_dir = os.path.join(dataset_dir, "train")
os.listdir(train_dir)
```
```
['unsupBow.feat',
'urls_pos.txt',
'labeledBow.feat',
'unsup',
'pos',
'urls_unsup.txt',
'urls_neg.txt',
'neg']
```
The `train` directory also contains additional folders which should be removed before creating the training set.
```python
remove_dir = os.path.join(train_dir, "unsup")
shutil.rmtree(remove_dir)
```
Next, create a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset.md.txt) using [`tf.keras.utils.text_dataset_from_directory`](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text_dataset_from_directory.md.txt). You can read more about using this utility in this [text classification tutorial](https://www.tensorflow.org/tutorials/keras/text_classification.md.txt).
Use the `train` directory to create the training and validation sets with a split of 20% for validation.
```python
batch_size = 1024
seed = 123
train_ds = tf.keras.utils.text_dataset_from_directory(
train_dir,
batch_size=batch_size,
validation_split=0.2,
subset="training",
seed=seed,
)
val_ds = tf.keras.utils.text_dataset_from_directory(
train_dir,
batch_size=batch_size,
validation_split=0.2,
subset="validation",
seed=seed,
)
```
```
Found 25000 files belonging to 2 classes.
Using 20000 files for training.
Found 25000 files belonging to 2 classes.
Using 5000 files for validation.
```
### Configure the dataset for performance
You can learn more about [`Dataset.cache`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset.md.txt#cache) and [`Dataset.prefetch`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset.md.txt#prefetch), as well as how to cache data to disk in the [data performance guide](https://www.tensorflow.org/guide/data_performance.md.txt).
```python
AUTOTUNE = tf.data.AUTOTUNE
train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
```
## Text preprocessing
Next, define the dataset preprocessing steps required for your sentiment classification model. Initialize a [`layers.TextVectorization`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/TextVectorization.md.txt) layer with the desired parameters to vectorize movie reviews. You can learn more about using this layer in the [Text Classification](https://www.tensorflow.org/tutorials/keras/text_classification.md.txt) tutorial.
```python
# Create a custom standardization function to strip HTML break tags '
'.
def custom_standardization(input_data):
lowercase = tf.strings.lower(input_data)
stripped_html = tf.strings.regex_replace(lowercase, "
", " ")
return tf.strings.regex_replace(
stripped_html, "[%s]" % re.escape(string.punctuation), ""
)
# Vocabulary size and number of words in a sequence.
vocab_size = 10000
sequence_length = 100
# Use the text vectorization layer to normalize, split, and map strings to
# integers. Note that the layer uses the custom standardization defined above.
# Set maximum_sequence length as all samples are not of the same length.
vectorize_layer = TextVectorization(
standardize=custom_standardization,
max_tokens=vocab_size,
output_mode="int",
output_sequence_length=sequence_length,
)
# Make a text-only dataset (no labels) and call `Dataset.adapt` to build the
# vocabulary.
text_ds = train_ds.map(lambda x, y: x)
vectorize_layer.adapt(text_ds)
```
## Create a classification model
Use the [Keras Sequential API](https://www.tensorflow.org/guide/keras/sequential_model.md.txt) to define the sentiment classification model.
```python
embedding_dim = 16
text_embedding = Embedding(vocab_size, embedding_dim, name="embedding")
```
```python
text_input = tf.keras.Sequential(
[vectorize_layer, text_embedding], name="text_input"
)
classifier_head = tf.keras.Sequential(
[GlobalAveragePooling1D(), Dense(16, activation="relu"), Dense(1)],
name="classifier_head",
)
model = tf.keras.Sequential([text_input, classifier_head])
```
## Compile and train the model
You will use [TensorBoard](https://www.tensorflow.org/tensorboard) to visualize metrics including loss and accuracy. Create a [`tf.keras.callbacks.TensorBoard`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard.md.txt).
```python
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
```
Compile and train the model using the `Adam` optimizer and `BinaryCrossentropy` loss.
```python
model.compile(
optimizer="adam",
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=["accuracy"],
)
```
```python
model.fit(
train_ds,
validation_data=val_ds,
epochs=15,
callbacks=[tensorboard_callback],
)
```
```
Epoch 1/15
20/20 [==============================] - 5s 192ms/step - loss: 0.6917 - accuracy: 0.5028 - val_loss: 0.6893 - val_accuracy: 0.4886
Epoch 2/15
20/20 [==============================] - 1s 51ms/step - loss: 0.6856 - accuracy: 0.5028 - val_loss: 0.6815 - val_accuracy: 0.4886
Epoch 3/15
20/20 [==============================] - 1s 51ms/step - loss: 0.6752 - accuracy: 0.5028 - val_loss: 0.6689 - val_accuracy: 0.4886
Epoch 4/15
20/20 [==============================] - 1s 50ms/step - loss: 0.6589 - accuracy: 0.5028 - val_loss: 0.6504 - val_accuracy: 0.4886
Epoch 5/15
20/20 [==============================] - 1s 50ms/step - loss: 0.6359 - accuracy: 0.5030 - val_loss: 0.6260 - val_accuracy: 0.4914
Epoch 6/15
20/20 [==============================] - 1s 50ms/step - loss: 0.6064 - accuracy: 0.5478 - val_loss: 0.5970 - val_accuracy: 0.5698
Epoch 7/15
20/20 [==============================] - 1s 50ms/step - loss: 0.5724 - accuracy: 0.6334 - val_loss: 0.5657 - val_accuracy: 0.6376
Epoch 8/15
20/20 [==============================] - 1s 50ms/step - loss: 0.5362 - accuracy: 0.7021 - val_loss: 0.5346 - val_accuracy: 0.6842
Epoch 9/15
20/20 [==============================] - 1s 50ms/step - loss: 0.5006 - accuracy: 0.7475 - val_loss: 0.5060 - val_accuracy: 0.7232
Epoch 10/15
20/20 [==============================] - 1s 51ms/step - loss: 0.4675 - accuracy: 0.7765 - val_loss: 0.4809 - val_accuracy: 0.7468
Epoch 11/15
20/20 [==============================] - 1s 50ms/step - loss: 0.4378 - accuracy: 0.7986 - val_loss: 0.4598 - val_accuracy: 0.7598
Epoch 12/15
20/20 [==============================] - 1s 50ms/step - loss: 0.4116 - accuracy: 0.8165 - val_loss: 0.4423 - val_accuracy: 0.7696
Epoch 13/15
20/20 [==============================] - 1s 50ms/step - loss: 0.3887 - accuracy: 0.8292 - val_loss: 0.4280 - val_accuracy: 0.7802
Epoch 14/15
20/20 [==============================] - 1s 50ms/step - loss: 0.3685 - accuracy: 0.8410 - val_loss: 0.4163 - val_accuracy: 0.7890
Epoch 15/15
20/20 [==============================] - 1s 50ms/step - loss: 0.3505 - accuracy: 0.8496 - val_loss: 0.4068 - val_accuracy: 0.7952
```
With this approach the model reaches a validation accuracy of around 85%
> [!NOTE]
> **Note:** Your results may be a bit different, depending on how weights were randomly initialized before training the embedding layer.
You can look into the model summary to learn more about each layer of the model.
```python
model.summary()
```
```
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
text_input (Sequential) (None, 100, 16) 160000
classifier_head (Sequentia (None, 1) 289
l)
=================================================================
Total params: 160289 (626.13 KB)
Trainable params: 160289 (626.13 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
```
Visualize the model metrics in TensorBoard.
```python
# docs_infra: no_execute
%load_ext tensorboard
%tensorboard --logdir logs
```
## Vocabulary remapping
Now you're going to update the vocabulary and continue with warm-started training.
First, get the base vocabulary and embedding matrix.
```python
embedding_weights_base = (
model.get_layer("text_input").get_layer("embedding").embeddings
)
vocab_base = vectorize_layer.get_vocabulary()
```
Define a new vectorization layer to generate a new bigger vocabulary
```python
# Vocabulary size and number of words in a sequence.
vocab_size_new = 10200
sequence_length = 100
vectorize_layer_new = TextVectorization(
standardize=custom_standardization,
max_tokens=vocab_size_new,
output_mode="int",
output_sequence_length=sequence_length,
)
# Make a text-only dataset (no labels) and call adapt to build the vocabulary.
text_ds = train_ds.map(lambda x, y: x)
vectorize_layer_new.adapt(text_ds)
# Get the new vocabulary
vocab_new = vectorize_layer_new.get_vocabulary()
```
```python
# View the new vocabulary tokens that weren't in `vocab_base`
set(vocab_base) ^ set(vocab_new)
```
```
{np.str_('bullying'),
np.str_('bumps'),
np.str_('canvas'),
np.str_('carole'),
np.str_('chains'),
np.str_('chairman'),
np.str_('checks'),
np.str_('coarse'),
np.str_('competitive'),
np.str_('component'),
np.str_('compound'),
np.str_('confirm'),
np.str_('contemplate'),
np.str_('coping'),
np.str_('corporations'),
np.str_('costuming'),
np.str_('counterpart'),
np.str_('crop'),
np.str_('custody'),
np.str_('cyborgs'),
np.str_('daft'),
np.str_('danced'),
np.str_('daphne'),
np.str_('darkest'),
np.str_('davids'),
np.str_('december'),
np.str_('declared'),
np.str_('defence'),
np.str_('delve'),
np.str_('demonstration'),
np.str_('dense'),
np.str_('denver'),
np.str_('devilish'),
np.str_('devious'),
np.str_('dickinson'),
np.str_('digs'),
np.str_('directorwriter'),
np.str_('download'),
np.str_('effortless'),
np.str_('electricity'),
np.str_('elliot'),
np.str_('enlightenment'),
np.str_('erratic'),
np.str_('exceedingly'),
np.str_('eyeballs'),
np.str_('fearless'),
np.str_('fenton'),
np.str_('fiennes'),
np.str_('filter'),
np.str_('fireworks'),
np.str_('flipping'),
np.str_('float'),
np.str_('foggy'),
np.str_('forgivable'),
np.str_('framework'),
np.str_('fulllength'),
np.str_('funds'),
np.str_('gamut'),
np.str_('geeks'),
np.str_('glee'),
np.str_('goo'),
np.str_('gripe'),
np.str_('hardest'),
np.str_('harmony'),
np.str_('henchman'),
np.str_('heritage'),
np.str_('hg'),
np.str_('hi'),
np.str_('hightech'),
np.str_('homework'),
np.str_('houston'),
np.str_('howards'),
np.str_('hunger'),
np.str_('imho'),
np.str_('immigrants'),
np.str_('improvised'),
np.str_('impulse'),
np.str_('inch'),
np.str_('interpret'),
np.str_('intimidating'),
np.str_('iowa'),
np.str_('jaffar'),
np.str_('jeep'),
np.str_('jock'),
np.str_('kriemhild'),
np.str_('kristofferson'),
np.str_('lassie'),
np.str_('laughoutloud'),
np.str_('lennon'),
np.str_('librarian'),
np.str_('liza'),
np.str_('locker'),
np.str_('lommel'),
np.str_('loren'),
np.str_('lowered'),
np.str_('marital'),
np.str_('martins'),
np.str_('mastroianni'),
np.str_('megan'),
np.str_('melt'),
np.str_('mischievous'),
np.str_('monstrosity'),
np.str_('monumental'),
np.str_('morse'),
np.str_('mostel'),
np.str_('muddy'),
np.str_('noah'),
np.str_('noirs'),
np.str_('nostril'),
np.str_('numbing'),
np.str_('occupation'),
np.str_('oceans'),
np.str_('onesided'),
np.str_('opus'),
np.str_('organ'),
np.str_('osullivan'),
np.str_('otoole'),
np.str_('overnight'),
np.str_('parisian'),
np.str_('partial'),
np.str_('patriotism'),
np.str_('pbs'),
np.str_('penchant'),
np.str_('penguin'),
np.str_('plotted'),
np.str_('powerfully'),
np.str_('pows'),
np.str_('practicing'),
np.str_('prehistoric'),
np.str_('prestigious'),
np.str_('prevalent'),
np.str_('prevents'),
np.str_('profits'),
np.str_('promotion'),
np.str_('puke'),
np.str_('pulse'),
np.str_('punchline'),
np.str_('quarters'),
np.str_('rainer'),
np.str_('ranting'),
np.str_('rapists'),
np.str_('rapture'),
np.str_('rarity'),
np.str_('rays'),
np.str_('recommending'),
np.str_('redeemed'),
np.str_('refuge'),
np.str_('refugee'),
np.str_('relates'),
np.str_('religions'),
np.str_('remaking'),
np.str_('renee'),
np.str_('reply'),
np.str_('restoration'),
np.str_('resurrection'),
np.str_('retreat'),
np.str_('retro'),
np.str_('rockets'),
np.str_('romano'),
np.str_('rooker'),
np.str_('rooted'),
np.str_('runtime'),
np.str_('sap'),
np.str_('scarred'),
np.str_('secluded'),
np.str_('selfabsorbed'),
np.str_('separation'),
np.str_('shattered'),
np.str_('shenanigans'),
np.str_('shootings'),
np.str_('shue'),
np.str_('silk'),
np.str_('sm'),
np.str_('soooo'),
np.str_('spoton'),
np.str_('sr'),
np.str_('staple'),
np.str_('stepfather'),
np.str_('stoic'),
np.str_('stud'),
np.str_('suite'),
np.str_('swanson'),
np.str_('sweetness'),
np.str_('sybil'),
np.str_('tease'),
np.str_('technological'),
np.str_('tensions'),
np.str_('theft'),
np.str_('therapist'),
np.str_('threats'),
np.str_('tin'),
np.str_('towel'),
np.str_('transform'),
np.str_('travelling'),
np.str_('troupe'),
np.str_('unremarkable'),
np.str_('unsatisfied'),
np.str_('untrue'),
np.str_('vertigo'),
np.str_('vic')}
```
Generate updated embeddings using the `keras.utils.warmstart_embedding_matrix` util.
```python
# Generate the updated embedding matrix
updated_embedding = tf.keras.utils.warmstart_embedding_matrix(
base_vocabulary=vocab_base,
new_vocabulary=vocab_new,
base_embeddings=embedding_weights_base,
new_embeddings_initializer="uniform",
)
# Update the model variable
updated_embedding_variable = tf.Variable(updated_embedding)
```
**OR**
If you have an embedding matrix which you would like to initialize the new embedding matrix with, use [`keras.initializers.Constant`](https://www.tensorflow.org/api_docs/python/tf/keras/initializers/Constant.md.txt) as new_embeddings initializer. Copy the following block to a code cell to try this out.
This would be helpful when you have a better embedding matrix initialization for new words in vocab.
# generate updated embedding matrix
new_embedding = np.random.rand(len(vocab_new), 16)
updated_embedding = tf.keras.utils.warmstart_embedding_matrix(
base_vocabulary=vocab_base,
new_vocabulary=vocab_new,
base_embeddings=embedding_weights_base,
new_embeddings_initializer=tf.keras.initializers.Constant(
new_embedding
)
)
# update model variable
updated_embedding_variable = tf.Variable(updated_embedding)
Verify if the embedding matrix's shape has changed to reflect the new vocabulary.
```python
updated_embedding_variable.shape
```
```
TensorShape([10200, 16])
```
Now that you have the updated embedding matrix, the next step is to update the layer weights.
```python
text_embedding_layer_new = Embedding(
vectorize_layer_new.vocabulary_size(), embedding_dim, name="embedding"
)
text_embedding_layer_new.build(input_shape=[None])
text_embedding_layer_new.embeddings.assign(updated_embedding)
text_input_new = tf.keras.Sequential(
[vectorize_layer_new, text_embedding_layer_new], name="text_input_new"
)
text_input_new.summary()
# Verify the shape of updated weights
# The new weights shape should reflect the new vocabulary size
text_input_new.get_layer("embedding").embeddings.shape
```
```
Model: "text_input_new"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
text_vectorization_1 (Text (None, 100) 0
Vectorization)
embedding (Embedding) (None, 100, 16) 163200
=================================================================
Total params: 163200 (637.50 KB)
Trainable params: 163200 (637.50 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
TensorShape([10200, 16])
```
Modify the model architecture to use the new text vectorization layer.
You can also load the model from a checkpoint and update the model architecture as shown below.
```python
warm_started_model = tf.keras.Sequential([text_input_new, classifier_head])
warm_started_model.summary()
```
```
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
text_input_new (Sequential (None, 100, 16) 163200
)
classifier_head (Sequentia (None, 1) 289
l)
=================================================================
Total params: 163489 (638.63 KB)
Trainable params: 163489 (638.63 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
```
You have successfully updated the model to accept a new vocabulary. The embedding layer is updated to map old vocabulary words to old embeddings and initialize embeddings for new vocabulary words to be learnt. The learned weights of the rest of the model will remain the same. The model is warm-started to continue to train from where it left off previously.
You can now verify that the remapping worked. Get the index of the vocabulary word "the" that is present both in base and new vocabulary and compare the embedding values. They should be equal.
```python
# New vocab words
base_vocab_index = vectorize_layer("the")[0]
new_vocab_index = vectorize_layer_new("the")[0]
print(
warm_started_model.get_layer("text_input_new").get_layer("embedding")(
new_vocab_index
)
== embedding_weights_base[base_vocab_index]
)
```
```
tf.Tensor(
[ True True True True True True True True True True True True
True True True True], shape=(16,), dtype=bool)
```
## Continue with warm-started training
Notice how the training is warm-started. The accuracy of first epoch is around 85%. This is close to the accuracy where the previous training ended.
```python
model.compile(
optimizer="adam",
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=["accuracy"],
)
model.fit(
train_ds,
validation_data=val_ds,
epochs=15,
callbacks=[tensorboard_callback],
)
```
```
Epoch 1/15
20/20 [==============================] - 5s 188ms/step - loss: 0.3362 - accuracy: 0.8554 - val_loss: 0.3994 - val_accuracy: 0.8014
Epoch 2/15
20/20 [==============================] - 1s 52ms/step - loss: 0.3220 - accuracy: 0.8622 - val_loss: 0.3936 - val_accuracy: 0.8040
Epoch 3/15
20/20 [==============================] - 1s 50ms/step - loss: 0.3098 - accuracy: 0.8680 - val_loss: 0.3888 - val_accuracy: 0.8080
Epoch 4/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2987 - accuracy: 0.8741 - val_loss: 0.3851 - val_accuracy: 0.8114
Epoch 5/15
20/20 [==============================] - 1s 51ms/step - loss: 0.2883 - accuracy: 0.8789 - val_loss: 0.3823 - val_accuracy: 0.8128
Epoch 6/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2787 - accuracy: 0.8832 - val_loss: 0.3802 - val_accuracy: 0.8154
Epoch 7/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2697 - accuracy: 0.8874 - val_loss: 0.3789 - val_accuracy: 0.8186
Epoch 8/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2611 - accuracy: 0.8913 - val_loss: 0.3782 - val_accuracy: 0.8208
Epoch 9/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2531 - accuracy: 0.8953 - val_loss: 0.3781 - val_accuracy: 0.8208
Epoch 10/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2454 - accuracy: 0.8988 - val_loss: 0.3785 - val_accuracy: 0.8200
Epoch 11/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2382 - accuracy: 0.9023 - val_loss: 0.3794 - val_accuracy: 0.8220
Epoch 12/15
20/20 [==============================] - 1s 51ms/step - loss: 0.2312 - accuracy: 0.9063 - val_loss: 0.3808 - val_accuracy: 0.8228
Epoch 13/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2245 - accuracy: 0.9096 - val_loss: 0.3826 - val_accuracy: 0.8222
Epoch 14/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2182 - accuracy: 0.9125 - val_loss: 0.3848 - val_accuracy: 0.8216
Epoch 15/15
20/20 [==============================] - 1s 50ms/step - loss: 0.2120 - accuracy: 0.9154 - val_loss: 0.3874 - val_accuracy: 0.8220
```
## Visualize warm-started training
```python
# docs_infra: no_execute
%reload_ext tensorboard
%tensorboard --logdir logs
```
## Next steps
In this tutorial you learned how to:
- Train a sentiment classification model from scratch on a small vocabulary dataset.
- Update the model architecture and warm start the embedding matrix when the vocabulary size changes.
- Continuously improve model accuracy with expanding datasets
To learn more about embeddings check out the [Word2Vec](https://www.tensorflow.org/tutorials/text/word2vec) and [Transformer model for language understanding](https://www.tensorflow.org/text/tutorials/transformer.md.txt) tutorials.