View on TensorFlow.org
|
Run in Google Colab
|
View source on GitHub
|
Download notebook
|
This text classification tutorial trains a recurrent neural network on the IMDB large movie review dataset for sentiment analysis.
Setup
!pip install -q -U tf-keras
import os
os.environ['TF_USE_LEGACY_KERAS'] = '1'
import numpy as np
import tensorflow_datasets as tfds
import tensorflow as tf
tfds.disable_progress_bar()
Import matplotlib and create a helper function to plot graphs:
import matplotlib.pyplot as plt
def plot_graphs(history, metric):
plt.plot(history.history[metric])
plt.plot(history.history['val_'+metric], '')
plt.xlabel("Epochs")
plt.ylabel(metric)
plt.legend([metric, 'val_'+metric])
Setup input pipeline
The IMDB large movie review dataset is a binary classification dataset—all the reviews have either a positive or negative sentiment.
Download the dataset using TFDS. See the loading text tutorial for details on how to load this sort of data manually.
dataset, info = tfds.load('imdb_reviews', with_info=True,
as_supervised=True)
train_dataset, test_dataset = dataset['train'], dataset['test']
train_dataset.element_spec
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR W0000 00:00:1795090879.571005 83264 dlopen_checker.cc:38] Could not load dynamic library 'libcusolver.so.11'; dlerror: /tmpfs/src/tf_docs_env/lib/python3.11/site-packages/tensorflow/python/platform/../../../nvidia/cusolver/lib/../../cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12 W0000 00:00:1795090879.571050 83264 dlopen_checker.cc:38] Could not load dynamic library 'libcusparse.so.12'; dlerror: /tmpfs/src/tf_docs_env/lib/python3.11/site-packages/tensorflow/python/platform/../../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12 W0000 00:00:1795090879.571054 83264 gpu_device.cc:2359] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. Skipping registering GPU devices... (TensorSpec(shape=(), dtype=tf.string, name=None), TensorSpec(shape=(), dtype=tf.int64, name=None))
Initially this returns a dataset of (text, label pairs):
for example, label in train_dataset.take(1):
print('text: ', example.numpy())
print('label: ', label.numpy())
text: b"This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie's ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. Maria Conchita Alonso appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor's like Christopher Walken's good name. I could barely sit through it." label: 0
Next shuffle the data for training and create batches of these (text, label) pairs:
BUFFER_SIZE = 10000
BATCH_SIZE = 64
train_dataset = train_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)
test_dataset = test_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)
for example, label in train_dataset.take(1):
print('texts: ', example.numpy()[:3])
print()
print('labels: ', label.numpy()[:3])
texts: [b'I have never seen a show as good as Full House. Full House puts all of the newer shows to shame, big time! Anyone who has never seen it, which I don\'t see how it is possible, should see it. It is a great show for anyone of any age. Full House will make you laugh, it will make you cry, it will amaze you. True, some people feel that there are some "cheesy" aspects to the show, but, the positive aspects out weigh all of the "cheesy" aspects. Full House ran it\'s first episode on September 22, 1987 entitled "Our Very First Show" and ran it\'s last episode on May 23, 1995 entitled "Michelle Rides Again Part II".<br /><br />The plot of the show is very believable. Danny Tanner (Bob Saget) losses his wife, Pam, in an accident involving a drunk driver. Danny has his brother in law, Jesse Katsopolis (John Stamos), which is Pam\'s younger brother, and Danny also brings in his best friend Joey Gladstone (Dave Coulier) to help him raise his three daughters. Danny\'s daughters are named DJ (Candice Cameron-Bure), Stephanie (Jodie Sweetin), and Michelle (Mary-Kate and Ashley Olsen).Joey and Jesse plan on moving in with Danny and his three girls for a few months just to help out and end up living with them for eight years; which is the number of years the show ran for.<br /><br />The following is a short description of some of the characters and the actor/actress who played him/her: John Stamos (Jesse): John Stamos is a great actor. He plays Jesse. Jesse is a rock star waiting to get his big break. In Full House, John Stamos does a great job portraying his character. He looked and played music like his idol, Elvis Presley.<br /><br />Bob Saget (Danny): Bob Saget is also a great actor. He looses his wife in car accident involving a drunk driver. He has to raise three girls without a having the girl\'s Mother. Bob Saget does a great job portraying a single parent who works full time and still has time to raise his three girls.<br /><br />Dave Coulier (Joey): One word can describe Dave Coulier, funny. He is great. Playing the character of Joey was perfect for him. He does a great job playing the stand-up comedian waiting for his big break.<br /><br />Candice Cameron-Bure (DJ): She is a tremendous actress. She plays the oldest sister, DJ which is short for Donna Jo. She is one of the best actresses I have ever seen. Her acting ability in Full House was very believable.<br /><br />Jodie Sweetin (Stephanie): Two simple words can describe Jodie Sweetin, incredibly amazing! I wish I could say every thing that I would like to say about Jodie, but, I would use up the 1,000 word maximum just on her. She got her start in a kids show called Mother Goose Stories and when she came to Full House, she blew the audience\'s and creator\'s mind. Her great looks and absolutely amazing acting ability helped to make the show the success that it was. According to Dave Coulier, Jodie was supposed to be the star of the show. It was supposed to be where she was going to get her big break. Jodie, at five years old when the show first aired, could hit every line perfectly. She showed great enthusiasm. Most young kids can\'t do this. As you can probably guess, Jodie Sweetin (Stephanie) is my favorite character in Full House.<br /><br />Mary-Kate and Ashley Olsen (Michelle): Great actress. Full House is where they got their start. They received the part of Michelle because they were the only babies who did not cry while in front of a camera.<br /><br />There are many more cast members that should be recognized. These are the original characters from when the show first went on the air in 1987.<br /><br />The only negative thing that I can say is how Full House became The Michelle Show towards the end. I think it was to focused on her towards the end. Especially when I think Jodie and Candice were much better at acting.<br /><br />Full House is a great show for everyone. It can teach you a lot. One of the biggest things it can teach you is that everyone can live a great life even if a tragedy, such as loosing a family member, occurs. Full House continues to attract new fans. With all this said, there is only a couple things left to say; Full House will never die, and, thank you, the cast of Full House, for giving everyone a show that they can enjoy.' b"May contain spoilers.<br /><br />I say that, but anyone savvy enough to be reading this can probably figure out every plot turn right from the start.<br /><br />This is not a movie that I liked. I didn't hate it in the way of some movies that insult your intelligence, but it all felt too predictable on its trudge to the requisite happy ending. There were funny bits along the way to be sure, but few were original. At least it didn't go for the gutter.<br /><br />Christina Applegate looks fresh, and Ben Affleck works hard. Their scenes together are actually the only redeeming feature. Everyone else is a cardboard cutout, including, surprisingly, James Gandolfini, who must have made this as a favor to someone.<br /><br />All in all, it's a harmless, but not inspiring, 90 minutes." b"This is one of the funniest movies I have seen. I watched it on DVD, and the disc does not have any special features, or even a menu, but that is not necessarily what I care about. <br /><br />I tend to judge movies on a case by case basis, depending on, among other things, if it is a big studio production or a smaller film. This is a smaller film and I am willing to forgive minor things. That said, I believe it has one of the most imaginative and original title sequences that I have seen.<br /><br />I enjoyed the acting of all of the major players. I especially enjoyed Til Schweiger and Alan Arkin. Alan Arkin has most of the funniest lines. The character portrayed by Claire Forlani might come across as unrealistic to some people, but I have personally known real people with emotional problems that very readily look at life's decisions as her character does. That helped me pick up the nuances where her hurts could come out through the veneer of her humor.<br /><br />This is not a movie for children, obviously, but it does NOT engage in gratuitous sex and nudity. There is quite a bit of adult language, though, but it can sometimes be very funny. (In particular, Alan Arkin's character, who can't even swear correctly.)<br /><br />Also watch for the cameos from known character actors."] labels: [1 0 1]
Create the text encoder
The raw text loaded by tfds needs to be processed before it can be used in a model. The simplest way to process text for training is using the TextVectorization layer. This layer has many capabilities, but this tutorial sticks to the default behavior.
Create the layer, and pass the dataset's text to the layer's .adapt method:
VOCAB_SIZE = 1000
encoder = tf.keras.layers.TextVectorization(
max_tokens=VOCAB_SIZE)
encoder.adapt(train_dataset.map(lambda text, label: text))
The .adapt method sets the layer's vocabulary. Here are the first 20 tokens. After the padding and unknown tokens they're sorted by frequency:
vocab = np.array(encoder.get_vocabulary())
vocab[:20]
array(['', '[UNK]', 'the', 'and', 'a', 'of', 'to', 'is', 'in', 'it', 'i',
'this', 'that', 'br', 'was', 'as', 'for', 'with', 'movie', 'but'],
dtype='<U14')
Once the vocabulary is set, the layer can encode text into indices. The tensors of indices are 0-padded to the longest sequence in the batch (unless you set a fixed output_sequence_length):
encoded_example = encoder(example)[:3].numpy()
encoded_example
array([[ 10, 26, 110, ..., 0, 0, 0],
[194, 1, 1, ..., 0, 0, 0],
[ 11, 7, 29, ..., 0, 0, 0]], shape=(3, 981))
With the default settings, the process is not completely reversible. There are three main reasons for that:
- The default value for
preprocessing.TextVectorization'sstandardizeargument is"lower_and_strip_punctuation". - The limited vocabulary size and lack of character-based fallback results in some unknown tokens.
for n in range(3):
print("Original: ", example[n].numpy())
print("Round-trip: ", " ".join(vocab[encoded_example[n]]))
print()
Original: b'I have never seen a show as good as Full House. Full House puts all of the newer shows to shame, big time! Anyone who has never seen it, which I don\'t see how it is possible, should see it. It is a great show for anyone of any age. Full House will make you laugh, it will make you cry, it will amaze you. True, some people feel that there are some "cheesy" aspects to the show, but, the positive aspects out weigh all of the "cheesy" aspects. Full House ran it\'s first episode on September 22, 1987 entitled "Our Very First Show" and ran it\'s last episode on May 23, 1995 entitled "Michelle Rides Again Part II".<br /><br />The plot of the show is very believable. Danny Tanner (Bob Saget) losses his wife, Pam, in an accident involving a drunk driver. Danny has his brother in law, Jesse Katsopolis (John Stamos), which is Pam\'s younger brother, and Danny also brings in his best friend Joey Gladstone (Dave Coulier) to help him raise his three daughters. Danny\'s daughters are named DJ (Candice Cameron-Bure), Stephanie (Jodie Sweetin), and Michelle (Mary-Kate and Ashley Olsen).Joey and Jesse plan on moving in with Danny and his three girls for a few months just to help out and end up living with them for eight years; which is the number of years the show ran for.<br /><br />The following is a short description of some of the characters and the actor/actress who played him/her: John Stamos (Jesse): John Stamos is a great actor. He plays Jesse. Jesse is a rock star waiting to get his big break. In Full House, John Stamos does a great job portraying his character. He looked and played music like his idol, Elvis Presley.<br /><br />Bob Saget (Danny): Bob Saget is also a great actor. He looses his wife in car accident involving a drunk driver. He has to raise three girls without a having the girl\'s Mother. Bob Saget does a great job portraying a single parent who works full time and still has time to raise his three girls.<br /><br />Dave Coulier (Joey): One word can describe Dave Coulier, funny. He is great. Playing the character of Joey was perfect for him. He does a great job playing the stand-up comedian waiting for his big break.<br /><br />Candice Cameron-Bure (DJ): She is a tremendous actress. She plays the oldest sister, DJ which is short for Donna Jo. She is one of the best actresses I have ever seen. Her acting ability in Full House was very believable.<br /><br />Jodie Sweetin (Stephanie): Two simple words can describe Jodie Sweetin, incredibly amazing! I wish I could say every thing that I would like to say about Jodie, but, I would use up the 1,000 word maximum just on her. She got her start in a kids show called Mother Goose Stories and when she came to Full House, she blew the audience\'s and creator\'s mind. Her great looks and absolutely amazing acting ability helped to make the show the success that it was. According to Dave Coulier, Jodie was supposed to be the star of the show. It was supposed to be where she was going to get her big break. Jodie, at five years old when the show first aired, could hit every line perfectly. She showed great enthusiasm. Most young kids can\'t do this. As you can probably guess, Jodie Sweetin (Stephanie) is my favorite character in Full House.<br /><br />Mary-Kate and Ashley Olsen (Michelle): Great actress. Full House is where they got their start. They received the part of Michelle because they were the only babies who did not cry while in front of a camera.<br /><br />There are many more cast members that should be recognized. These are the original characters from when the show first went on the air in 1987.<br /><br />The only negative thing that I can say is how Full House became The Michelle Show towards the end. I think it was to focused on her towards the end. Especially when I think Jodie and Candice were much better at acting.<br /><br />Full House is a great show for everyone. It can teach you a lot. One of the biggest things it can teach you is that everyone can live a great life even if a tragedy, such as loosing a family member, occurs. Full House continues to attract new fans. With all this said, there is only a couple things left to say; Full House will never die, and, thank you, the cast of Full House, for giving everyone a show that they can enjoy.' Round-trip: i have never seen a show as good as full house full house [UNK] all of the [UNK] shows to shame big time anyone who has never seen it which i dont see how it is possible should see it it is a great show for anyone of any age full house will make you laugh it will make you [UNK] it will [UNK] you true some people feel that there are some cheesy [UNK] to the show but the [UNK] [UNK] out [UNK] all of the cheesy [UNK] full house [UNK] its first episode on [UNK] [UNK] [UNK] [UNK] our very first show and [UNK] its last episode on may [UNK] [UNK] [UNK] [UNK] [UNK] again part [UNK] br the plot of the show is very believable [UNK] [UNK] [UNK] [UNK] [UNK] his wife [UNK] in an [UNK] [UNK] a [UNK] [UNK] [UNK] has his brother in [UNK] [UNK] [UNK] john [UNK] which is [UNK] [UNK] brother and [UNK] also brings in his best friend [UNK] [UNK] [UNK] [UNK] to help him [UNK] his three [UNK] [UNK] [UNK] are named [UNK] [UNK] [UNK] [UNK] [UNK] [UNK] and [UNK] [UNK] and [UNK] [UNK] and [UNK] [UNK] on moving in with [UNK] and his three girls for a few [UNK] just to help out and end up living with them for [UNK] years which is the number of years the show [UNK] [UNK] br the [UNK] is a short [UNK] of some of the characters and the [UNK] who played [UNK] john [UNK] [UNK] john [UNK] is a great actor he plays [UNK] [UNK] is a rock star [UNK] to get his big [UNK] in full house john [UNK] does a great job [UNK] his character he looked and played music like his [UNK] [UNK] [UNK] br [UNK] [UNK] [UNK] [UNK] [UNK] is also a great actor he [UNK] his wife in car [UNK] [UNK] a [UNK] [UNK] he has to [UNK] three girls without a having the girls mother [UNK] [UNK] does a great job [UNK] a single [UNK] who works full time and still has time to [UNK] his three [UNK] br [UNK] [UNK] [UNK] one word can [UNK] [UNK] [UNK] funny he is great playing the character of [UNK] was perfect for him he does a great job playing the [UNK] [UNK] [UNK] for his big [UNK] br [UNK] [UNK] [UNK] she is a [UNK] actress she plays the [UNK] sister [UNK] which is short for [UNK] [UNK] she is one of the best [UNK] i have ever seen her acting [UNK] in full house was very [UNK] br [UNK] [UNK] [UNK] two simple words can [UNK] [UNK] [UNK] incredibly amazing i wish i could say every thing that i would like to say about [UNK] but i would use up the [UNK] word [UNK] just on her she got her start in a kids show called mother [UNK] stories and when she came to full house she [UNK] the [UNK] and [UNK] mind her great looks and absolutely amazing acting [UNK] [UNK] to make the show the [UNK] that it was [UNK] to [UNK] [UNK] [UNK] was supposed to be the star of the show it was supposed to be where she was going to get her big [UNK] [UNK] at five years old when the show first [UNK] could hit every line perfectly she [UNK] great [UNK] most young kids cant do this as you can probably guess [UNK] [UNK] [UNK] is my favorite character in full [UNK] br [UNK] and [UNK] [UNK] [UNK] great actress full house is where they got their start they [UNK] the part of [UNK] because they were the only [UNK] who did not [UNK] while in [UNK] of a [UNK] br there are many more cast [UNK] that should be [UNK] these are the original characters from when the show first went on the air in [UNK] br the only [UNK] thing that i can say is how full house became the [UNK] show towards the end i think it was to [UNK] on her towards the end especially when i think [UNK] and [UNK] were much better at [UNK] br full house is a great show for everyone it can [UNK] you a lot one of the [UNK] things it can [UNK] you is that everyone can live a great life even if a [UNK] such as [UNK] a family [UNK] [UNK] full house [UNK] to [UNK] new fans with all this said there is only a couple things left to say full house will never die and [UNK] you the cast of full house for giving everyone a show that they can enjoy Original: b"May contain spoilers.<br /><br />I say that, but anyone savvy enough to be reading this can probably figure out every plot turn right from the start.<br /><br />This is not a movie that I liked. I didn't hate it in the way of some movies that insult your intelligence, but it all felt too predictable on its trudge to the requisite happy ending. There were funny bits along the way to be sure, but few were original. At least it didn't go for the gutter.<br /><br />Christina Applegate looks fresh, and Ben Affleck works hard. Their scenes together are actually the only redeeming feature. Everyone else is a cardboard cutout, including, surprisingly, James Gandolfini, who must have made this as a favor to someone.<br /><br />All in all, it's a harmless, but not inspiring, 90 minutes." Round-trip: may [UNK] [UNK] br i say that but anyone [UNK] enough to be reading this can probably figure out every plot turn right from the [UNK] br this is not a movie that i liked i didnt hate it in the way of some movies that [UNK] your [UNK] but it all felt too predictable on its [UNK] to the [UNK] happy ending there were funny [UNK] along the way to be sure but few were original at least it didnt go for the [UNK] br [UNK] [UNK] looks [UNK] and [UNK] [UNK] works hard their scenes together are actually the only [UNK] feature everyone else is a [UNK] [UNK] including [UNK] james [UNK] who must have made this as a [UNK] to [UNK] br all in all its a [UNK] but not [UNK] [UNK] minutes Original: b"This is one of the funniest movies I have seen. I watched it on DVD, and the disc does not have any special features, or even a menu, but that is not necessarily what I care about. <br /><br />I tend to judge movies on a case by case basis, depending on, among other things, if it is a big studio production or a smaller film. This is a smaller film and I am willing to forgive minor things. That said, I believe it has one of the most imaginative and original title sequences that I have seen.<br /><br />I enjoyed the acting of all of the major players. I especially enjoyed Til Schweiger and Alan Arkin. Alan Arkin has most of the funniest lines. The character portrayed by Claire Forlani might come across as unrealistic to some people, but I have personally known real people with emotional problems that very readily look at life's decisions as her character does. That helped me pick up the nuances where her hurts could come out through the veneer of her humor.<br /><br />This is not a movie for children, obviously, but it does NOT engage in gratuitous sex and nudity. There is quite a bit of adult language, though, but it can sometimes be very funny. (In particular, Alan Arkin's character, who can't even swear correctly.)<br /><br />Also watch for the cameos from known character actors." Round-trip: this is one of the [UNK] movies i have seen i watched it on dvd and the [UNK] does not have any special features or even a [UNK] but that is not [UNK] what i care about br br i [UNK] to [UNK] movies on a case by case [UNK] [UNK] on among other things if it is a big [UNK] production or a [UNK] film this is a [UNK] film and i am [UNK] to [UNK] [UNK] things that said i believe it has one of the most [UNK] and original title sequences that i have [UNK] br i enjoyed the acting of all of the major [UNK] i especially enjoyed [UNK] [UNK] and [UNK] [UNK] [UNK] [UNK] has most of the [UNK] lines the character portrayed by [UNK] [UNK] might come across as [UNK] to some people but i have [UNK] known real people with emotional problems that very [UNK] look at [UNK] [UNK] as her character does that [UNK] me [UNK] up the [UNK] where her [UNK] could come out through the [UNK] of her [UNK] br this is not a movie for children obviously but it does not [UNK] in [UNK] sex and [UNK] there is quite a bit of [UNK] [UNK] though but it can sometimes be very funny in particular [UNK] [UNK] character who cant even [UNK] [UNK] br also watch for the [UNK] from known character actors
Create the model

Above is a diagram of the model.
This model can be build as a
tf.keras.Sequential.The first layer is the
encoder, which converts the text to a sequence of token indices.After the encoder is an embedding layer. An embedding layer stores one vector per word. When called, it converts the sequences of word indices to sequences of vectors. These vectors are trainable. After training (on enough data), words with similar meanings often have similar vectors.
This index-lookup is much more efficient than the equivalent operation of passing a one-hot encoded vector through a
tf.keras.layers.Denselayer.A recurrent neural network (RNN) processes sequence input by iterating through the elements. RNNs pass the outputs from one timestep to their input on the next timestep.
The
tf.keras.layers.Bidirectionalwrapper can also be used with an RNN layer. This propagates the input forward and backwards through the RNN layer and then concatenates the final output.The main advantage of a bidirectional RNN is that the signal from the beginning of the input doesn't need to be processed all the way through every timestep to affect the output.
The main disadvantage of a bidirectional RNN is that you can't efficiently stream predictions as words are being added to the end.
After the RNN has converted the sequence to a single vector the two
layers.Densedo some final processing, and convert from this vector representation to a single logit as the classification output.
The code to implement this is below:
model = tf.keras.Sequential([
encoder,
tf.keras.layers.Embedding(
input_dim=len(encoder.get_vocabulary()),
output_dim=64,
# Use masking to handle the variable sequence lengths
mask_zero=True),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(1)
])
Please note that Keras sequential model is used here since all the layers in the model only have single input and produce single output. In case you want to use stateful RNN layer, you might want to build your model with Keras functional API or model subclassing so that you can retrieve and reuse the RNN layer states. Please check Keras RNN guide for more details.
The embedding layer uses masking to handle the varying sequence-lengths. All the layers after the Embedding support masking:
print([layer.supports_masking for layer in model.layers])
[False, True, True, True, True]
To confirm that this works as expected, evaluate a sentence twice. First, alone so there's no padding to mask:
# predict on a sample text without padding.
sample_text = ('The movie was cool. The animation and the graphics '
'were out of this world. I would recommend this movie.')
predictions = model.predict(np.array([sample_text]))
print(predictions[0])
1/1 [==============================] - 2s 2s/step [0.00550823]
Now, evaluate it again in a batch with a longer sentence. The result should be identical:
# predict on a sample text with padding
padding = "the " * 2000
predictions = model.predict(np.array([sample_text, padding]))
print(predictions[0])
1/1 [==============================] - 0s 139ms/step [0.00550823]
Compile the Keras model to configure the training process:
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
optimizer=tf.keras.optimizers.Adam(1e-4),
metrics=['accuracy'])
Train the model
history = model.fit(train_dataset, epochs=10,
validation_data=test_dataset,
validation_steps=30)
Epoch 1/10 391/391 [==============================] - 191s 470ms/step - loss: 0.6440 - accuracy: 0.5683 - val_loss: 0.5954 - val_accuracy: 0.5594 Epoch 2/10 391/391 [==============================] - 180s 460ms/step - loss: 0.4540 - accuracy: 0.7752 - val_loss: 0.4047 - val_accuracy: 0.8188 Epoch 3/10 391/391 [==============================] - 178s 455ms/step - loss: 0.3642 - accuracy: 0.8405 - val_loss: 0.3634 - val_accuracy: 0.8417 Epoch 4/10 391/391 [==============================] - 178s 454ms/step - loss: 0.3323 - accuracy: 0.8564 - val_loss: 0.3464 - val_accuracy: 0.8339 Epoch 5/10 391/391 [==============================] - 177s 451ms/step - loss: 0.3167 - accuracy: 0.8646 - val_loss: 0.3305 - val_accuracy: 0.8427 Epoch 6/10 391/391 [==============================] - 177s 452ms/step - loss: 0.3107 - accuracy: 0.8652 - val_loss: 0.3260 - val_accuracy: 0.8552 Epoch 7/10 391/391 [==============================] - 178s 453ms/step - loss: 0.3079 - accuracy: 0.8668 - val_loss: 0.3219 - val_accuracy: 0.8589 Epoch 8/10 391/391 [==============================] - 179s 456ms/step - loss: 0.3040 - accuracy: 0.8699 - val_loss: 0.3386 - val_accuracy: 0.8375 Epoch 9/10 391/391 [==============================] - 176s 449ms/step - loss: 0.3071 - accuracy: 0.8682 - val_loss: 0.3416 - val_accuracy: 0.8620 Epoch 10/10 391/391 [==============================] - 177s 451ms/step - loss: 0.3006 - accuracy: 0.8715 - val_loss: 0.3219 - val_accuracy: 0.8573
test_loss, test_acc = model.evaluate(test_dataset)
print('Test Loss:', test_loss)
print('Test Accuracy:', test_acc)
391/391 [==============================] - 53s 134ms/step - loss: 0.3166 - accuracy: 0.8552 Test Loss: 0.31655046343803406 Test Accuracy: 0.8551999926567078
plt.figure(figsize=(16, 8))
plt.subplot(1, 2, 1)
plot_graphs(history, 'accuracy')
plt.ylim(None, 1)
plt.subplot(1, 2, 2)
plot_graphs(history, 'loss')
plt.ylim(0, None)
(0.0, 0.6611749708652497)

Run a prediction on a new sentence:
If the prediction is >= 0.0, it is positive else it is negative.
sample_text = ('The movie was cool. The animation and the graphics '
'were out of this world. I would recommend this movie.')
predictions = model.predict(np.array([sample_text]))
1/1 [==============================] - 2s 2s/step
Stack two or more LSTM layers
Keras recurrent layers have two available modes that are controlled by the return_sequences constructor argument:
If
Falseit returns only the last output for each input sequence (a 2D tensor of shape (batch_size, output_features)). This is the default, used in the previous model.If
Truethe full sequences of successive outputs for each timestep is returned (a 3D tensor of shape(batch_size, timesteps, output_features)).
Here is what the flow of information looks like with return_sequences=True:

The interesting thing about using an RNN with return_sequences=True is that the output still has 3-axes, like the input, so it can be passed to another RNN layer, like this:
model = tf.keras.Sequential([
encoder,
tf.keras.layers.Embedding(len(encoder.get_vocabulary()), 64, mask_zero=True),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1)
])
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
optimizer=tf.keras.optimizers.Adam(1e-4),
metrics=['accuracy'])
history = model.fit(train_dataset, epochs=10,
validation_data=test_dataset,
validation_steps=30)
Epoch 1/10 391/391 [==============================] - 357s 878ms/step - loss: 0.6255 - accuracy: 0.5879 - val_loss: 0.4441 - val_accuracy: 0.7901 Epoch 2/10 391/391 [==============================] - 334s 854ms/step - loss: 0.3853 - accuracy: 0.8282 - val_loss: 0.3571 - val_accuracy: 0.8333 Epoch 3/10 391/391 [==============================] - 337s 859ms/step - loss: 0.3421 - accuracy: 0.8503 - val_loss: 0.3346 - val_accuracy: 0.8422 Epoch 4/10 391/391 [==============================] - 336s 857ms/step - loss: 0.3221 - accuracy: 0.8615 - val_loss: 0.3230 - val_accuracy: 0.8521 Epoch 5/10 391/391 [==============================] - 334s 852ms/step - loss: 0.3125 - accuracy: 0.8659 - val_loss: 0.3209 - val_accuracy: 0.8594 Epoch 6/10 391/391 [==============================] - 337s 860ms/step - loss: 0.3062 - accuracy: 0.8685 - val_loss: 0.3193 - val_accuracy: 0.8615 Epoch 7/10 391/391 [==============================] - 334s 853ms/step - loss: 0.3035 - accuracy: 0.8703 - val_loss: 0.3149 - val_accuracy: 0.8609 Epoch 8/10 391/391 [==============================] - 333s 851ms/step - loss: 0.3007 - accuracy: 0.8706 - val_loss: 0.3228 - val_accuracy: 0.8630 Epoch 9/10 391/391 [==============================] - 332s 847ms/step - loss: 0.2985 - accuracy: 0.8715 - val_loss: 0.3212 - val_accuracy: 0.8677 Epoch 10/10 391/391 [==============================] - 332s 848ms/step - loss: 0.2958 - accuracy: 0.8730 - val_loss: 0.3156 - val_accuracy: 0.8505
test_loss, test_acc = model.evaluate(test_dataset)
print('Test Loss:', test_loss)
print('Test Accuracy:', test_acc)
391/391 [==============================] - 92s 236ms/step - loss: 0.3170 - accuracy: 0.8528 Test Loss: 0.3170185387134552 Test Accuracy: 0.852840006351471
# predict on a sample text without padding.
sample_text = ('The movie was not good. The animation and the graphics '
'were terrible. I would not recommend this movie.')
predictions = model.predict(np.array([sample_text]))
print(predictions)
1/1 [==============================] - 5s 5s/step [[-1.7038605]]
plt.figure(figsize=(16, 6))
plt.subplot(1, 2, 1)
plot_graphs(history, 'accuracy')
plt.subplot(1, 2, 2)
plot_graphs(history, 'loss')

Check out other existing recurrent layers such as GRU layers.
If you're interested in building custom RNNs, see the Keras RNN Guide.
View on TensorFlow.org
Run in Google Colab
View source on GitHub
Download notebook