Missed TensorFlow World? Check out the recap. Learn more

Text classification with an RNN

View on TensorFlow.org Run in Google Colab View source on GitHub Download notebook

This text classification tutorial trains a recurrent neural network on the IMDB large movie review dataset for sentiment analysis.

Setup

from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow_datasets as tfds
import tensorflow as tf

Import matplotlib and create a helper function to plot graphs:

import matplotlib.pyplot as plt

def plot_graphs(history, string):
  plt.plot(history.history[string])
  plt.plot(history.history['val_'+string], '')
  plt.xlabel("Epochs")
  plt.ylabel(string)
  plt.legend([string, 'val_'+string])
  plt.show()

Setup input pipeline

The IMDB large movie review dataset is a binary classification dataset—all the reviews have either a positive or negative sentiment.

Download the dataset using TFDS.

dataset, info = tfds.load('imdb_reviews/subwords8k', with_info=True,
                          as_supervised=True)
train_dataset, test_dataset = dataset['train'], dataset['test']

The dataset info includes the encoder (a tfds.features.text.SubwordTextEncoder).

encoder = info.features['text'].encoder
print ('Vocabulary size: {}'.format(encoder.vocab_size))
Vocabulary size: 8185

This text encoder will reversibly encode any string, falling back to byte-encoding if necessary.

sample_string = 'Hello TensorFlow.'

encoded_string = encoder.encode(sample_string)
print ('Encoded string is {}'.format(encoded_string))

original_string = encoder.decode(encoded_string)
print ('The original string: "{}"'.format(original_string))
Encoded string is [4025, 222, 6307, 2327, 4043, 2120, 7975]
The original string: "Hello TensorFlow."
assert original_string == sample_string
for index in encoded_string:
  print ('{} ----> {}'.format(index, encoder.decode([index])))
4025 ----> Hell
222 ----> o 
6307 ----> Ten
2327 ----> sor
4043 ----> Fl
2120 ----> ow
7975 ----> .

Prepare the data for training

Next create batches of these encoded strings. Use the padded_batch method to zero-pad the sequences to the length of the longest string n the batch:

BUFFER_SIZE = 10000
BATCH_SIZE = 64
train_dataset = train_dataset.shuffle(BUFFER_SIZE)
train_dataset = train_dataset.padded_batch(BATCH_SIZE, train_dataset.output_shapes)

test_dataset = test_dataset.padded_batch(BATCH_SIZE, test_dataset.output_shapes)

Create the model

Build a tf.keras.Sequential model and start with an embedding layer. An embedding layer stores one vector per word. When called, it converts the sequences of word indices to sequences of vectors. These vectors are trainable. After training (on enough data), words with similar meanings often have similar vectors.

This index-lookup is much more efficient than the equivalent operation of passing a one-hot encoded vector through a tf.keras.layers.Dense layer.

A recurrent neural network (RNN) processes sequence input by iterating through the elements. RNNs pass the outputs from one timestep to their input—and then to the next.

The tf.keras.layers.Bidirectional wrapper can also be used with an RNN layer. This propagates the input forward and backwards through the RNN layer and then concatenates the output. This helps the RNN to learn long range dependencies.

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(encoder.vocab_size, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

Compile the Keras model to configure the training process:

model.compile(loss='binary_crossentropy',
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['accuracy'])

Train the model

history = model.fit(train_dataset, epochs=10,
                    validation_data=test_dataset, 
                    validation_steps=30)
Epoch 1/10
391/391 [==============================] - 55s 141ms/step - loss: 0.6530 - accuracy: 0.5983 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 2/10
391/391 [==============================] - 48s 124ms/step - loss: 0.3648 - accuracy: 0.8521 - val_loss: 0.3552 - val_accuracy: 0.8484
Epoch 3/10
391/391 [==============================] - 48s 124ms/step - loss: 0.2560 - accuracy: 0.9024 - val_loss: 0.3304 - val_accuracy: 0.8609
Epoch 4/10
391/391 [==============================] - 48s 123ms/step - loss: 0.2153 - accuracy: 0.9230 - val_loss: 0.3230 - val_accuracy: 0.8615
Epoch 5/10
391/391 [==============================] - 48s 123ms/step - loss: 0.1887 - accuracy: 0.9346 - val_loss: 0.3435 - val_accuracy: 0.8615
Epoch 6/10
391/391 [==============================] - 48s 123ms/step - loss: 0.1711 - accuracy: 0.9416 - val_loss: 0.3324 - val_accuracy: 0.8641
Epoch 7/10
391/391 [==============================] - 48s 122ms/step - loss: 0.1452 - accuracy: 0.9524 - val_loss: 0.3489 - val_accuracy: 0.8531
Epoch 8/10
391/391 [==============================] - 48s 122ms/step - loss: 0.1326 - accuracy: 0.9570 - val_loss: 0.4295 - val_accuracy: 0.8589
Epoch 9/10
391/391 [==============================] - 48s 123ms/step - loss: 0.1168 - accuracy: 0.9643 - val_loss: 0.4225 - val_accuracy: 0.8578
Epoch 10/10
391/391 [==============================] - 48s 122ms/step - loss: 0.1139 - accuracy: 0.9648 - val_loss: 0.4547 - val_accuracy: 0.8568
test_loss, test_acc = model.evaluate(test_dataset)

print('Test Loss: {}'.format(test_loss))
print('Test Accuracy: {}'.format(test_acc))
391/391 [==============================] - 18s 46ms/step - loss: 0.4621 - accuracy: 0.8581
Test Loss: 0.46211091787232766
Test Accuracy: 0.8581200242042542

The above model does not mask the padding applied to the sequences. This can lead to skew if trained on padded sequences and test on un-padded sequences. Ideally you would use masking to avoid this, but as you can see below it only have a small effect on the output.

If the prediction is >= 0.5, it is positive else it is negative.

def pad_to_size(vec, size):
  zeros = [0] * (size - len(vec))
  vec.extend(zeros)
  return vec
def sample_predict(sentence, pad):
  encoded_sample_pred_text = encoder.encode(sample_pred_text)

  if pad:
    encoded_sample_pred_text = pad_to_size(encoded_sample_pred_text, 64)
  encoded_sample_pred_text = tf.cast(encoded_sample_pred_text, tf.float32)
  predictions = model.predict(tf.expand_dims(encoded_sample_pred_text, 0))

  return (predictions)
# predict on a sample text without padding.

sample_pred_text = ('The movie was cool. The animation and the graphics '
                    'were out of this world. I would recommend this movie.')
predictions = sample_predict(sample_pred_text, pad=False)
print (predictions)
[[0.41101742]]
# predict on a sample text with padding

sample_pred_text = ('The movie was cool. The animation and the graphics '
                    'were out of this world. I would recommend this movie.')
predictions = sample_predict(sample_pred_text, pad=True)
print (predictions)
[[0.4186573]]
plot_graphs(history, 'accuracy')

png

plot_graphs(history, 'loss')

png

Stack two or more LSTM layers

Keras recurrent layers have two available modes that are controlled by the return_sequences constructor argument:

  • Return either the full sequences of successive outputs for each timestep (a 3D tensor of shape (batch_size, timesteps, output_features)).
  • Return only the last output for each input sequence (a 2D tensor of shape (batch_size, output_features)).
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(encoder.vocab_size, 64),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64,  return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy',
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['accuracy'])
history = model.fit(train_dataset, epochs=10,
                    validation_data=test_dataset,
                    validation_steps=30)
Epoch 1/10
391/391 [==============================] - 89s 228ms/step - loss: 0.6491 - accuracy: 0.5914 - val_loss: 0.0000e+00 - val_accuracy: 0.0000e+00
Epoch 2/10
391/391 [==============================] - 81s 207ms/step - loss: 0.3512 - accuracy: 0.8623 - val_loss: 0.3416 - val_accuracy: 0.8505
Epoch 3/10
391/391 [==============================] - 81s 207ms/step - loss: 0.2710 - accuracy: 0.9048 - val_loss: 0.3342 - val_accuracy: 0.8672
Epoch 4/10
391/391 [==============================] - 81s 208ms/step - loss: 0.2143 - accuracy: 0.9285 - val_loss: 0.3488 - val_accuracy: 0.8766
Epoch 5/10
391/391 [==============================] - 81s 206ms/step - loss: 0.1845 - accuracy: 0.9406 - val_loss: 0.3586 - val_accuracy: 0.8625
Epoch 6/10
391/391 [==============================] - 81s 207ms/step - loss: 0.1615 - accuracy: 0.9513 - val_loss: 0.3647 - val_accuracy: 0.8646
Epoch 7/10
391/391 [==============================] - 81s 207ms/step - loss: 0.1420 - accuracy: 0.9575 - val_loss: 0.4124 - val_accuracy: 0.8693
Epoch 8/10
391/391 [==============================] - 81s 207ms/step - loss: 0.1175 - accuracy: 0.9677 - val_loss: 0.4462 - val_accuracy: 0.8594
Epoch 9/10
391/391 [==============================] - 80s 206ms/step - loss: 0.1034 - accuracy: 0.9734 - val_loss: 0.4958 - val_accuracy: 0.8510
Epoch 10/10
391/391 [==============================] - 82s 209ms/step - loss: 0.0923 - accuracy: 0.9777 - val_loss: 0.5143 - val_accuracy: 0.8573
test_loss, test_acc = model.evaluate(test_dataset)

print('Test Loss: {}'.format(test_loss))
print('Test Accuracy: {}'.format(test_acc))
391/391 [==============================] - 34s 86ms/step - loss: 0.5185 - accuracy: 0.8543
Test Loss: 0.5184965819844505
Test Accuracy: 0.8543199896812439
# predict on a sample text without padding.

sample_pred_text = ('The movie was not good. The animation and the graphics '
                    'were terrible. I would not recommend this movie.')
predictions = sample_predict(sample_pred_text, pad=False)
print (predictions)
[[0.05746633]]
# predict on a sample text with padding

sample_pred_text = ('The movie was not good. The animation and the graphics '
                    'were terrible. I would not recommend this movie.')
predictions = sample_predict(sample_pred_text, pad=True)
print (predictions)
[[0.01921545]]
plot_graphs(history, 'accuracy')

png

plot_graphs(history, 'loss')

png

Check out other existing recurrent layers such as GRU layers.

If you're interestied in building custom RNNs, see the Keras RNN Guide.