 View on TensorFlow.org
|
 Run in Google Colab
|
 View on GitHub
|
 Download notebook
|
 See TF Hub model
|
TF-Hub is a platform to share machine learning expertise packaged in reusable resources, notably pre-trained modules. In this tutorial, we will use a TF-Hub text embedding module to train a simple sentiment classifier with a reasonable baseline accuracy. We will then submit the predictions to Kaggle.
For more detailed tutorial on text classification with TF-Hub and further steps for improving the accuracy, take a look at Text classification with TF-Hub.
Setup
pip install -q kaggle
import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import zipfile
from sklearn import model_selection
2021-07-29 12:03:27.042705: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Since this tutorial will be using a dataset from Kaggle, it requires creating an API Token for your Kaggle account, and uploading it to the Colab environment.
import os
import pathlib
# Upload the API token.
def get_kaggle():
try:
import kaggle
return kaggle
except OSError:
pass
token_file = pathlib.Path("~/.kaggle/kaggle.json").expanduser()
token_file.parent.mkdir(exist_ok=True, parents=True)
try:
from google.colab import files
except ImportError:
raise ValueError("Could not find kaggle token.")
uploaded = files.upload()
token_content = uploaded.get('kaggle.json', None)
if token_content:
token_file.write_bytes(token_content)
token_file.chmod(0o600)
else:
raise ValueError('Need a file named "kaggle.json"')
import kaggle
return kaggle
kaggle = get_kaggle()
Getting started
Data
We will try to solve the Sentiment Analysis on Movie Reviews task from Kaggle. The dataset consists of syntactic subphrases of the Rotten Tomatoes movie reviews. The task is to label the phrases as negative or positive on the scale from 1 to 5.
You must accept the competition rules before you can use the API to download the data.
SENTIMENT_LABELS = [
"negative", "somewhat negative", "neutral", "somewhat positive", "positive"
]
# Add a column with readable values representing the sentiment.
def add_readable_labels_column(df, sentiment_value_column):
df["SentimentLabel"] = df[sentiment_value_column].replace(
range(5), SENTIMENT_LABELS)
# Download data from Kaggle and create a DataFrame.
def load_data_from_zip(path):
with zipfile.ZipFile(path, "r") as zip_ref:
name = zip_ref.namelist()[0]
with zip_ref.open(name) as zf:
return pd.read_csv(zf, sep="\t", index_col=0)
# The data does not come with a validation set so we'll create one from the
# training set.
def get_data(competition, train_file, test_file, validation_set_ratio=0.1):
data_path = pathlib.Path("data")
kaggle.api.competition_download_files(competition, data_path)
competition_path = (data_path/competition)
competition_path.mkdir(exist_ok=True, parents=True)
competition_zip_path = competition_path.with_suffix(".zip")
with zipfile.ZipFile(competition_zip_path, "r") as zip_ref:
zip_ref.extractall(competition_path)
train_df = load_data_from_zip(competition_path/train_file)
test_df = load_data_from_zip(competition_path/test_file)
# Add a human readable label.
add_readable_labels_column(train_df, "Sentiment")
# We split by sentence ids, because we don't want to have phrases belonging
# to the same sentence in both training and validation set.
train_indices, validation_indices = model_selection.train_test_split(
np.unique(train_df["SentenceId"]),
test_size=validation_set_ratio,
random_state=0)
validation_df = train_df[train_df["SentenceId"].isin(validation_indices)]
train_df = train_df[train_df["SentenceId"].isin(train_indices)]
print("Split the training data into %d training and %d validation examples." %
(len(train_df), len(validation_df)))
return train_df, validation_df, test_df
train_df, validation_df, test_df = get_data(
"sentiment-analysis-on-movie-reviews",
"train.tsv.zip", "test.tsv.zip")
Split the training data into 140315 training and 15745 validation examples.
train_df.head(20)
Training an Model
class MyModel(tf.keras.Model):
def __init__(self, hub_url):
super().__init__()
self.hub_url = hub_url
self.embed = hub.load(self.hub_url).signatures['default']
self.sequential = tf.keras.Sequential([
tf.keras.layers.Dense(500),
tf.keras.layers.Dense(100),
tf.keras.layers.Dense(5),
])
def call(self, inputs):
phrases = inputs['Phrase'][:,0]
embedding = 5*self.embed(phrases)['default']
return self.sequential(embedding)
def get_config(self):
return {"hub_url":self.hub_url}
model = MyModel("https://tfhub.dev/google/nnlm-en-dim128/1")
model.compile(
loss = tf.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=tf.optimizers.Adam(),
metrics = [tf.keras.metrics.SparseCategoricalAccuracy(name="accuracy")])
2021-07-29 12:03:30.932413: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1 2021-07-29 12:03:31.599608: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:31.600632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:00:05.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0 coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s 2021-07-29 12:03:31.600666: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-07-29 12:03:31.603814: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 2021-07-29 12:03:31.603923: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 2021-07-29 12:03:31.604901: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10 2021-07-29 12:03:31.605239: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10 2021-07-29 12:03:31.606007: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11 2021-07-29 12:03:31.606749: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11 2021-07-29 12:03:31.606940: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8 2021-07-29 12:03:31.607054: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:31.608070: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:31.608989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0 2021-07-29 12:03:31.609810: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-07-29 12:03:31.610368: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:31.611264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: pciBusID: 0000:00:05.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0 coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s 2021-07-29 12:03:31.611366: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:31.612288: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:31.613182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0 2021-07-29 12:03:31.613225: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0 2021-07-29 12:03:32.256560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-07-29 12:03:32.256599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0 2021-07-29 12:03:32.256607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N 2021-07-29 12:03:32.256882: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:32.257874: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:32.258906: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-29 12:03:32.259892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14646 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:05.0, compute capability: 7.0) 2021-07-29 12:03:42.372065: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2) 2021-07-29 12:03:42.372607: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2000179999 Hz
history = model.fit(x=dict(train_df), y=train_df['Sentiment'],
validation_data=(dict(validation_df), validation_df['Sentiment']),
epochs = 25)
Epoch 1/25 2021-07-29 12:03:43.728806: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11 56/4385 [..............................] - ETA: 12s - loss: 1.3175 - accuracy: 0.4676 2021-07-29 12:03:44.147082: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11 4385/4385 [==============================] - 15s 3ms/step - loss: 1.0238 - accuracy: 0.5865 - val_loss: 0.9978 - val_accuracy: 0.5947 Epoch 2/25 4385/4385 [==============================] - 14s 3ms/step - loss: 0.9990 - accuracy: 0.5944 - val_loss: 0.9865 - val_accuracy: 0.5912 Epoch 3/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9955 - accuracy: 0.5970 - val_loss: 0.9840 - val_accuracy: 0.5917 Epoch 4/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9929 - accuracy: 0.5973 - val_loss: 0.9865 - val_accuracy: 0.5899 Epoch 5/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9910 - accuracy: 0.5990 - val_loss: 0.9881 - val_accuracy: 0.5932 Epoch 6/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9902 - accuracy: 0.5983 - val_loss: 0.9785 - val_accuracy: 0.6001 Epoch 7/25 4385/4385 [==============================] - 14s 3ms/step - loss: 0.9895 - accuracy: 0.5988 - val_loss: 0.9783 - val_accuracy: 0.5963 Epoch 8/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9884 - accuracy: 0.5993 - val_loss: 0.9826 - val_accuracy: 0.5917 Epoch 9/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9886 - accuracy: 0.5980 - val_loss: 0.9874 - val_accuracy: 0.5915 Epoch 10/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9882 - accuracy: 0.5983 - val_loss: 0.9786 - val_accuracy: 0.5988 Epoch 11/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9878 - accuracy: 0.5992 - val_loss: 0.9778 - val_accuracy: 0.5964 Epoch 12/25 4385/4385 [==============================] - 14s 3ms/step - loss: 0.9872 - accuracy: 0.5987 - val_loss: 0.9840 - val_accuracy: 0.5899 Epoch 13/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9875 - accuracy: 0.5991 - val_loss: 0.9778 - val_accuracy: 0.5950 Epoch 14/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9872 - accuracy: 0.5990 - val_loss: 0.9846 - val_accuracy: 0.5903 Epoch 15/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9866 - accuracy: 0.5998 - val_loss: 0.9775 - val_accuracy: 0.5966 Epoch 16/25 4385/4385 [==============================] - 14s 3ms/step - loss: 0.9868 - accuracy: 0.5993 - val_loss: 0.9879 - val_accuracy: 0.5903 Epoch 17/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9863 - accuracy: 0.5992 - val_loss: 0.9816 - val_accuracy: 0.5928 Epoch 18/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9865 - accuracy: 0.6007 - val_loss: 0.9831 - val_accuracy: 0.5945 Epoch 19/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9861 - accuracy: 0.6001 - val_loss: 0.9766 - val_accuracy: 0.5994 Epoch 20/25 4385/4385 [==============================] - 14s 3ms/step - loss: 0.9860 - accuracy: 0.5996 - val_loss: 0.9792 - val_accuracy: 0.5971 Epoch 21/25 4385/4385 [==============================] - 14s 3ms/step - loss: 0.9860 - accuracy: 0.6005 - val_loss: 0.9840 - val_accuracy: 0.5928 Epoch 22/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9858 - accuracy: 0.5999 - val_loss: 0.9807 - val_accuracy: 0.6001 Epoch 23/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9859 - accuracy: 0.5991 - val_loss: 0.9806 - val_accuracy: 0.5932 Epoch 24/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9857 - accuracy: 0.6000 - val_loss: 0.9752 - val_accuracy: 0.5993 Epoch 25/25 4385/4385 [==============================] - 15s 3ms/step - loss: 0.9855 - accuracy: 0.6004 - val_loss: 0.9792 - val_accuracy: 0.6034
Prediction
Run predictions for the validation set and training set.
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
[<matplotlib.lines.Line2D at 0x7f70dc7ec990>]
train_eval_result = model.evaluate(dict(train_df), train_df['Sentiment'])
validation_eval_result = model.evaluate(dict(validation_df), validation_df['Sentiment'])
print(f"Training set accuracy: {train_eval_result[1]}")
print(f"Validation set accuracy: {validation_eval_result[1]}")
4385/4385 [==============================] - 14s 3ms/step - loss: 0.9850 - accuracy: 0.6022 493/493 [==============================] - 1s 2ms/step - loss: 0.9792 - accuracy: 0.6034 Training set accuracy: 0.6021594405174255 Validation set accuracy: 0.6034296751022339
Confusion matrix
Another very interesting statistic, especially for multiclass problems, is the confusion matrix. The confusion matrix allows visualization of the proportion of correctly and incorrectly labelled examples. We can easily see how much our classifier is biased and whether the distribution of labels makes sense. Ideally the largest fraction of predictions should be distributed along the diagonal.
predictions = model.predict(dict(validation_df))
predictions = tf.argmax(predictions, axis=-1)
predictions
<tf.Tensor: shape=(15745,), dtype=int64, numpy=array([1, 1, 2, ..., 2, 2, 2])>
cm = tf.math.confusion_matrix(validation_df['Sentiment'], predictions)
cm = cm/cm.numpy().sum(axis=1)[:, tf.newaxis]
sns.heatmap(
cm, annot=True,
xticklabels=SENTIMENT_LABELS,
yticklabels=SENTIMENT_LABELS)
plt.xlabel("Predicted")
plt.ylabel("True")
Text(32.99999999999999, 0.5, 'True')
We can easily submit the predictions back to Kaggle by pasting the following code to a code cell and executing it:
test_predictions = model.predict(dict(test_df))
test_predictions = np.argmax(test_predictions, axis=-1)
result_df = test_df.copy()
result_df["Predictions"] = test_predictions
result_df.to_csv(
"predictions.csv",
columns=["Predictions"],
header=["Sentiment"])
kaggle.api.competition_submit("predictions.csv", "Submitted from Colab",
"sentiment-analysis-on-movie-reviews")
After submitting, check the leaderboard to see how you did.