![]() |
![]() |
![]() |
![]() |
![]() |
The CORD-19 Swivel text embedding module from TF-Hub (https://tfhub.dev/tensorflow/cord-19/swivel-128d/3) was built to support researchers analyzing natural languages text related to COVID-19. These embeddings were trained on the titles, authors, abstracts, body texts, and reference titles of articles in the CORD-19 dataset.
In this colab we will:
- Analyze semantically similar words in the embedding space
- Train a classifier on the SciCite dataset using the CORD-19 embeddings
Setup
import functools
import itertools
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import pandas as pd
import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow_hub as hub
from tqdm import trange
Analyze the embeddings
Let's start off by analyzing the embedding by calculating and plotting a correlation matrix between different terms. If the embedding learned to successfully capture the meaning of different words, the embedding vectors of semantically similar words should be close together. Let's take a look at some COVID-19 related terms.
# Use the inner product between two embedding vectors as the similarity measure
def plot_correlation(labels, features):
corr = np.inner(features, features)
corr /= np.max(corr)
sns.heatmap(corr, xticklabels=labels, yticklabels=labels)
# Generate embeddings for some terms
queries = [
# Related viruses
'coronavirus', 'SARS', 'MERS',
# Regions
'Italy', 'Spain', 'Europe',
# Symptoms
'cough', 'fever', 'throat'
]
module = hub.load('https://tfhub.dev/tensorflow/cord-19/swivel-128d/3')
embeddings = module(queries)
plot_correlation(queries, embeddings)
We can see that the embedding successfully captured the meaning of the different terms. Each word is similar to the other words of its cluster (i.e. "coronavirus" highly correlates with "SARS" and "MERS"), while they are different from terms of other clusters (i.e. the similarity between "SARS" and "Spain" is close to 0).
Now let's see how we can use these embeddings to solve a specific task.
SciCite: Citation Intent Classification
This section shows how one can use the embedding for downstream tasks such as text classification. We'll use the SciCite dataset from TensorFlow Datasets to classify citation intents in academic papers. Given a sentence with a citation from an academic paper, classify whether the main intent of the citation is as background information, use of methods, or comparing results.
builder = tfds.builder(name='scicite')
builder.download_and_prepare()
train_data, validation_data, test_data = builder.as_dataset(
split=('train', 'validation', 'test'),
as_supervised=True)
Let's take a look at a few labeled examples from the training set
NUM_EXAMPLES = 10
TEXT_FEATURE_NAME = builder.info.supervised_keys[0]
LABEL_NAME = builder.info.supervised_keys[1]
def label2str(numeric_label):
m = builder.info.features[LABEL_NAME].names
return m[numeric_label]
data = next(iter(train_data.batch(NUM_EXAMPLES)))
pd.DataFrame({
TEXT_FEATURE_NAME: [ex.numpy().decode('utf8') for ex in data[0]],
LABEL_NAME: [label2str(x) for x in data[1]]
})
Training a citaton intent classifier
We'll train a classifier on the SciCite dataset using Keras. Let's build a model which use the CORD-19 embeddings with a classification layer on top.
Hyperparameters
EMBEDDING = 'https://tfhub.dev/tensorflow/cord-19/swivel-128d/3'
TRAINABLE_MODULE = False
hub_layer = hub.KerasLayer(EMBEDDING, input_shape=[],
dtype=tf.string, trainable=TRAINABLE_MODULE)
model = tf.keras.Sequential()
model.add(hub_layer)
model.add(tf.keras.layers.Dense(3))
model.summary()
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= keras_layer (KerasLayer) (None, 128) 17301632 dense (Dense) (None, 3) 387 ================================================================= Total params: 17302019 (132.00 MB) Trainable params: 387 (1.51 KB) Non-trainable params: 17301632 (132.00 MB) _________________________________________________________________
Train and evaluate the model
Let's train and evaluate the model to see the performance on the SciCite task
EPOCHS = 35
BATCH_SIZE = 32
history = model.fit(train_data.shuffle(10000).batch(BATCH_SIZE),
epochs=EPOCHS,
validation_data=validation_data.batch(BATCH_SIZE),
verbose=1)
Epoch 1/35 257/257 [==============================] - 3s 5ms/step - loss: 0.8760 - accuracy: 0.6140 - val_loss: 0.7403 - val_accuracy: 0.7227 Epoch 2/35 257/257 [==============================] - 1s 4ms/step - loss: 0.6880 - accuracy: 0.7259 - val_loss: 0.6526 - val_accuracy: 0.7555 Epoch 3/35 257/257 [==============================] - 1s 4ms/step - loss: 0.6211 - accuracy: 0.7575 - val_loss: 0.6142 - val_accuracy: 0.7598 Epoch 4/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5870 - accuracy: 0.7720 - val_loss: 0.5962 - val_accuracy: 0.7609 Epoch 5/35 257/257 [==============================] - 2s 4ms/step - loss: 0.5675 - accuracy: 0.7812 - val_loss: 0.5818 - val_accuracy: 0.7609 Epoch 6/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5553 - accuracy: 0.7856 - val_loss: 0.5755 - val_accuracy: 0.7675 Epoch 7/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5461 - accuracy: 0.7889 - val_loss: 0.5697 - val_accuracy: 0.7707 Epoch 8/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5394 - accuracy: 0.7934 - val_loss: 0.5633 - val_accuracy: 0.7762 Epoch 9/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5339 - accuracy: 0.7955 - val_loss: 0.5620 - val_accuracy: 0.7773 Epoch 10/35 257/257 [==============================] - 2s 4ms/step - loss: 0.5294 - accuracy: 0.7950 - val_loss: 0.5582 - val_accuracy: 0.7740 Epoch 11/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5263 - accuracy: 0.7948 - val_loss: 0.5565 - val_accuracy: 0.7773 Epoch 12/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5233 - accuracy: 0.7941 - val_loss: 0.5547 - val_accuracy: 0.7784 Epoch 13/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5206 - accuracy: 0.7958 - val_loss: 0.5538 - val_accuracy: 0.7740 Epoch 14/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5185 - accuracy: 0.7942 - val_loss: 0.5513 - val_accuracy: 0.7740 Epoch 15/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5164 - accuracy: 0.7968 - val_loss: 0.5509 - val_accuracy: 0.7740 Epoch 16/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5149 - accuracy: 0.7968 - val_loss: 0.5498 - val_accuracy: 0.7784 Epoch 17/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5135 - accuracy: 0.7988 - val_loss: 0.5483 - val_accuracy: 0.7762 Epoch 18/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5115 - accuracy: 0.7990 - val_loss: 0.5480 - val_accuracy: 0.7817 Epoch 19/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5101 - accuracy: 0.7980 - val_loss: 0.5489 - val_accuracy: 0.7806 Epoch 20/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5090 - accuracy: 0.7985 - val_loss: 0.5463 - val_accuracy: 0.7849 Epoch 21/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5081 - accuracy: 0.7980 - val_loss: 0.5477 - val_accuracy: 0.7806 Epoch 22/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5073 - accuracy: 0.7995 - val_loss: 0.5473 - val_accuracy: 0.7817 Epoch 23/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5064 - accuracy: 0.7977 - val_loss: 0.5483 - val_accuracy: 0.7817 Epoch 24/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5053 - accuracy: 0.7989 - val_loss: 0.5496 - val_accuracy: 0.7806 Epoch 25/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5046 - accuracy: 0.7997 - val_loss: 0.5496 - val_accuracy: 0.7849 Epoch 26/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5043 - accuracy: 0.7996 - val_loss: 0.5455 - val_accuracy: 0.7882 Epoch 27/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5033 - accuracy: 0.8001 - val_loss: 0.5458 - val_accuracy: 0.7849 Epoch 28/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5029 - accuracy: 0.8021 - val_loss: 0.5463 - val_accuracy: 0.7882 Epoch 29/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5021 - accuracy: 0.8014 - val_loss: 0.5455 - val_accuracy: 0.7860 Epoch 30/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5020 - accuracy: 0.8024 - val_loss: 0.5445 - val_accuracy: 0.7893 Epoch 31/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5008 - accuracy: 0.8017 - val_loss: 0.5439 - val_accuracy: 0.7849 Epoch 32/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5007 - accuracy: 0.8011 - val_loss: 0.5434 - val_accuracy: 0.7838 Epoch 33/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5002 - accuracy: 0.8028 - val_loss: 0.5464 - val_accuracy: 0.7849 Epoch 34/35 257/257 [==============================] - 1s 4ms/step - loss: 0.5000 - accuracy: 0.8019 - val_loss: 0.5480 - val_accuracy: 0.7860 Epoch 35/35 257/257 [==============================] - 1s 4ms/step - loss: 0.4992 - accuracy: 0.8014 - val_loss: 0.5447 - val_accuracy: 0.7882
from matplotlib import pyplot as plt
def display_training_curves(training, validation, title, subplot):
if subplot%10==1: # set up the subplots on the first call
plt.subplots(figsize=(10,10), facecolor='#F0F0F0')
plt.tight_layout()
ax = plt.subplot(subplot)
ax.set_facecolor('#F8F8F8')
ax.plot(training)
ax.plot(validation)
ax.set_title('model '+ title)
ax.set_ylabel(title)
ax.set_xlabel('epoch')
ax.legend(['train', 'valid.'])
display_training_curves(history.history['accuracy'], history.history['val_accuracy'], 'accuracy', 211)
display_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 212)
/tmpfs/tmp/ipykernel_50997/4094752860.py:6: MatplotlibDeprecationWarning: Auto-removal of overlapping axes is deprecated since 3.6 and will be removed two minor releases later; explicitly call ax.remove() as needed. ax = plt.subplot(subplot)
Evaluate the model
And let's see how the model performs. Two values will be returned. Loss (a number which represents our error, lower values are better), and accuracy.
results = model.evaluate(test_data.batch(512), verbose=2)
for name, value in zip(model.metrics_names, results):
print('%s: %.3f' % (name, value))
4/4 - 0s - loss: 0.5376 - accuracy: 0.7897 - 336ms/epoch - 84ms/step loss: 0.538 accuracy: 0.790
We can see that the loss quickly decreases while especially the accuracy rapidly increases. Let's plot some examples to check how the prediction relates to the true labels:
prediction_dataset = next(iter(test_data.batch(20)))
prediction_texts = [ex.numpy().decode('utf8') for ex in prediction_dataset[0]]
prediction_labels = [label2str(x) for x in prediction_dataset[1]]
predictions = [
label2str(x) for x in np.argmax(model.predict(prediction_texts), axis=-1)]
pd.DataFrame({
TEXT_FEATURE_NAME: prediction_texts,
LABEL_NAME: prediction_labels,
'prediction': predictions
})
1/1 [==============================] - 0s 124ms/step
We can see that for this random sample, the model predicts the correct label most of the times, indicating that it can embed scientific sentences pretty well.
What's next?
Now that you've gotten to know a bit more about the CORD-19 Swivel embeddings from TF-Hub, we encourage you to participate in the CORD-19 Kaggle competition to contribute to gaining scientific insights from COVID-19 related academic texts.
- Participate in the CORD-19 Kaggle Challenge
- Learn more about the COVID-19 Open Research Dataset (CORD-19)
- See documentation and more about the TF-Hub embeddings at https://tfhub.dev/tensorflow/cord-19/swivel-128d/3
- Explore the CORD-19 embedding space with the TensorFlow Embedding Projector