Copyright 2019 Google LLC
 View on TensorFlow.org
|
 Run in Google Colab
|
 View source on GitHub
|
Overview
Graph regularization is a specific technique under the broader paradigm of Neural Graph Learning (Bui et al., 2018). The core idea is to train neural network models with a graph-regularized objective, harnessing both labeled and unlabeled data.
In this tutorial, we will explore the use of graph regularization to classify documents that form a natural (organic) graph.
The general recipe for creating a graph-regularized model using the Neural Structured Learning (NSL) framework is as follows:
- Generate training data from the input graph and sample features. Nodes in the graph correspond to samples and edges in the graph correspond to similarity between pairs of samples. The resulting training data will contain neighbor features in addition to the original node features.
- Create a neural network as a base model using the
Kerassequential, functional, or subclass API. - Wrap the base model with the
GraphRegularizationwrapper class, which is provided by the NSL framework, to create a new graphKerasmodel. This new model will include a graph regularization loss as the regularization term in its training objective. - Train and evaluate the graph
Kerasmodel.
Setup
- Install TensorFlow 2.2.x to create an interactive development environment with eager execution.
- Install the Neural Structured Learning package.
pip install --quiet tf-nightly==2.2.0.dev20200119
ERROR: tensorflow 2.1.0 has requirement gast==0.2.2, but you'll have gast 0.3.2 which is incompatible.
pip install --quiet neural-structured-learning
Dependencies and imports
from __future__ import absolute_import, division, print_function, unicode_literals
import neural_structured_learning as nsl
import tensorflow as tf
# Resets notebook state
tf.keras.backend.clear_session()
print("Version: ", tf.__version__)
print("Eager mode: ", tf.executing_eagerly())
print("GPU is", "available" if tf.test.is_gpu_available() else "NOT AVAILABLE")
Version: 2.2.0-dev20200119
Eager mode: True
WARNING:tensorflow:From <ipython-input-4-0436f456f879>:12: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
GPU is NOT AVAILABLE
Cora dataset
The Cora dataset is a citation graph where nodes represent machine learning papers and edges represent citations between pairs of papers. The task involved is document classification where the goal is to categorize each paper into one of 7 categories. In other words, this is a multi-class classification problem with 7 classes.
Graph
The original graph is directed. However, for the purpose of this example, we consider the undirected version of this graph. So, if paper A cites paper B, we also consider paper B to have cited A. Although this is not necessarily true, in this example, we consider citations as a proxy for similarity, which is usually a commutative property.
Features
Each paper in the input effectively contains 2 features:
Words: A dense, multi-hot bag-of-words representation of the text in the paper. The vocabulary for the Cora dataset contains 1433 unique words. So, the length of this feature is 1433, and the value at position 'i' is 0/1 indicating whether word 'i' in the vocabulary exists in the given paper or not.
Label: A single integer representing the class ID (category) of the paper.
Download the Cora dataset
wget --quiet -P /tmp https://linqs-data.soe.ucsc.edu/public/lbc/cora.tgztar -C /tmp -xvzf /tmp/cora.tgz
cora/ cora/README cora/cora.content cora/cora.cites
Convert the Cora data to the NSL format
In order to preprocess the Cora dataset and convert it to the format required by Neural Structured Learning, we will run the 'preprocess_cora_dataset.py' script, which is included in the NSL github repository. This script does the following:
- Generate neighbor features using the original node features and the graph.
- Generate train and test data splits containing
tf.train.Exampleinstances. - Persist the resulting train and test data in the
TFRecordformat.
!wget https://raw.githubusercontent.com/tensorflow/neural-structured-learning/master/neural_structured_learning/examples/preprocess/cora/preprocess_cora_dataset.py
!python preprocess_cora_dataset.py \
--input_cora_content=/tmp/cora/cora.content \
--input_cora_graph=/tmp/cora/cora.cites \
--max_nbrs=5 \
--output_train_data=/tmp/cora/train_merged_examples.tfr \
--output_test_data=/tmp/cora/test_examples.tfr
--2020-02-07 04:07:22-- https://raw.githubusercontent.com/tensorflow/neural-structured-learning/master/neural_structured_learning/examples/preprocess/cora/preprocess_cora_dataset.py Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 11419 (11K) [text/plain] Saving to: ‘preprocess_cora_dataset.py’ preprocess_cora_dat 100%[===================>] 11.15K --.-KB/s in 0s 2020-02-07 04:07:22 (120 MB/s) - ‘preprocess_cora_dataset.py’ saved [11419/11419] Reading graph file: /tmp/cora/cora.cites... Done reading 5429 edges from: /tmp/cora/cora.cites (0.01 seconds). Making all edges bi-directional... Done (0.01 seconds). Total graph nodes: 2708 Joining seed and neighbor tf.train.Examples with graph edges... Done creating and writing 2155 merged tf.train.Examples (1.36 seconds). Out-degree histogram: [(1, 386), (2, 468), (3, 452), (4, 309), (5, 540)] Output training data written to TFRecord file: /tmp/cora/train_merged_examples.tfr. Output test data written to TFRecord file: /tmp/cora/test_examples.tfr. Total running time: 0.04 minutes.
Global variables
The file paths to the train and test data are based on the command line flag values used to invoke the 'preprocess_cora_dataset.py' script above.
### Experiment dataset
TRAIN_DATA_PATH = '/tmp/cora/train_merged_examples.tfr'
TEST_DATA_PATH = '/tmp/cora/test_examples.tfr'
### Constants used to identify neighbor features in the input.
NBR_FEATURE_PREFIX = 'NL_nbr_'
NBR_WEIGHT_SUFFIX = '_weight'
Hyperparameters
We will use an instance of HParams to include various hyperparameters and
constants used for training and evaluation. We briefly describe each of them
below:
num_classes: There are a total 7 different classes
max_seq_length: This is the size of the vocabulary and all instances in the input have a dense multi-hot, bag-of-words representation. In other words, a value of 1 for a word indicates that the word is present in the input and a value of 0 indicates that it is not.
distance_type: This is the distance metric used to regularize the sample with its neighbors.
graph_regularization_multiplier: This controls the relative weight of the graph regularization term in the overall loss function.
num_neighbors: The number of neighbors used for graph regularization. This value has to be less than or equal to the
max_nbrscommand-line argument used above when runningpreprocess_cora_dataset.py.num_fc_units: The number of fully connected layers in our neural network.
train_epochs: The number of training epochs.
batch_size: Batch size used for training and evaluation.
dropout_rate: Controls the rate of dropout following each fully connected layer
eval_steps: The number of batches to process before deeming evaluation is complete. If set to
None, all instances in the test set are evaluated.
class HParams(object):
"""Hyperparameters used for training."""
def __init__(self):
### dataset parameters
self.num_classes = 7
self.max_seq_length = 1433
### neural graph learning parameters
self.distance_type = nsl.configs.DistanceType.L2
self.graph_regularization_multiplier = 0.1
self.num_neighbors = 1
### model architecture
self.num_fc_units = [50, 50]
### training parameters
self.train_epochs = 100
self.batch_size = 128
self.dropout_rate = 0.5
### eval parameters
self.eval_steps = None # All instances in the test set are evaluated.
HPARAMS = HParams()
Load train and test data
As described earlier in this notebook, the input training and test data have
been created by the 'preprocess_cora_dataset.py'. We will load them into two
tf.data.Dataset objects -- one for train and one for test.
In the input layer of our model, we will extract not just the 'words' and the
'label' features from each sample, but also corresponding neighbor features
based on the hparams.num_neighbors value. Instances with fewer neighbors than
hparams.num_neighbors will be assigned dummy values for those non-existent
neighbor features.
def parse_example(example_proto):
"""Extracts relevant fields from the `example_proto`.
Args:
example_proto: An instance of `tf.train.Example`.
Returns:
A pair whose first value is a dictionary containing relevant features
and whose second value contains the ground truth label.
"""
# The 'words' feature is a multi-hot, bag-of-words representation of the
# original raw text. A default value is required for examples that don't
# have the feature.
feature_spec = {
'words':
tf.io.FixedLenFeature([HPARAMS.max_seq_length],
tf.int64,
default_value=tf.constant(
0,
dtype=tf.int64,
shape=[HPARAMS.max_seq_length])),
'label':
tf.io.FixedLenFeature((), tf.int64, default_value=-1),
}
# We also extract corresponding neighbor features in a similar manner to
# the features above.
for i in range(HPARAMS.num_neighbors):
nbr_feature_key = '{}{}_{}'.format(NBR_FEATURE_PREFIX, i, 'words')
nbr_weight_key = '{}{}{}'.format(NBR_FEATURE_PREFIX, i, NBR_WEIGHT_SUFFIX)
feature_spec[nbr_feature_key] = tf.io.FixedLenFeature(
[HPARAMS.max_seq_length],
tf.int64,
default_value=tf.constant(
0, dtype=tf.int64, shape=[HPARAMS.max_seq_length]))
# We assign a default value of 0.0 for the neighbor weight so that
# graph regularization is done on samples based on their exact number
# of neighbors. In other words, non-existent neighbors are discounted.
feature_spec[nbr_weight_key] = tf.io.FixedLenFeature(
[1], tf.float32, default_value=tf.constant([0.0]))
features = tf.io.parse_single_example(example_proto, feature_spec)
label = features.pop('label')
return features, label
def make_dataset(file_path, training=False):
"""Creates a `tf.data.TFRecordDataset`.
Args:
file_path: Name of the file in the `.tfrecord` format containing
`tf.train.Example` objects.
training: Boolean indicating if we are in training mode.
Returns:
An instance of `tf.data.TFRecordDataset` containing the `tf.train.Example`
objects.
"""
dataset = tf.data.TFRecordDataset([file_path])
if training:
dataset = dataset.shuffle(10000)
dataset = dataset.map(parse_example)
dataset = dataset.batch(HPARAMS.batch_size)
return dataset
train_dataset = make_dataset(TRAIN_DATA_PATH, training=True)
test_dataset = make_dataset(TEST_DATA_PATH)
Let's peek into the train dataset to look at its contents.
for feature_batch, label_batch in train_dataset.take(1):
print('Feature list:', list(feature_batch.keys()))
print('Batch of inputs:', feature_batch['words'])
nbr_feature_key = '{}{}_{}'.format(NBR_FEATURE_PREFIX, 0, 'words')
nbr_weight_key = '{}{}{}'.format(NBR_FEATURE_PREFIX, 0, NBR_WEIGHT_SUFFIX)
print('Batch of neighbor inputs:', feature_batch[nbr_feature_key])
print('Batch of neighbor weights:',
tf.reshape(feature_batch[nbr_weight_key], [-1]))
print('Batch of labels:', label_batch)
Feature list: ['NL_nbr_0_words', 'words', 'NL_nbr_0_weight'] Batch of inputs: tf.Tensor( [[0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] ... [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0]], shape=(128, 1433), dtype=int64) Batch of neighbor inputs: tf.Tensor( [[0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] ... [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0]], shape=(128, 1433), dtype=int64) Batch of neighbor weights: tf.Tensor( [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.], shape=(128,), dtype=float32) Batch of labels: tf.Tensor( [6 2 0 0 0 2 3 2 3 6 1 5 4 2 1 1 4 2 4 3 4 4 1 1 5 3 2 3 6 2 6 2 6 2 6 6 2 4 3 2 2 3 6 1 2 3 1 2 5 2 3 1 3 4 2 0 0 6 2 6 2 5 6 1 6 1 3 0 6 1 5 3 2 2 5 1 2 5 0 0 2 1 2 2 3 6 5 2 2 2 3 3 3 2 2 2 1 6 2 5 2 2 6 3 4 2 1 2 6 3 1 1 2 2 1 2 6 3 1 0 4 6 2 2 1 4 0 2], shape=(128,), dtype=int64)
Let's peek into the test dataset to look at its contents.
for feature_batch, label_batch in test_dataset.take(1):
print('Feature list:', list(feature_batch.keys()))
print('Batch of inputs:', feature_batch['words'])
nbr_feature_key = '{}{}_{}'.format(NBR_FEATURE_PREFIX, 0, 'words')
nbr_weight_key = '{}{}{}'.format(NBR_FEATURE_PREFIX, 0, NBR_WEIGHT_SUFFIX)
print('Batch of neighbor inputs:', feature_batch[nbr_feature_key])
print('Batch of neighbor weights:',
tf.reshape(feature_batch[nbr_weight_key], [-1]))
print('Batch of labels:', label_batch)
Feature list: ['NL_nbr_0_words', 'words', 'NL_nbr_0_weight'] Batch of inputs: tf.Tensor( [[0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] ... [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0]], shape=(128, 1433), dtype=int64) Batch of neighbor inputs: tf.Tensor( [[0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] ... [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0]], shape=(128, 1433), dtype=int64) Batch of neighbor weights: tf.Tensor( [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.], shape=(128,), dtype=float32) Batch of labels: tf.Tensor( [2 3 6 2 2 2 0 2 0 2 6 3 2 2 3 4 4 1 2 0 3 2 2 2 2 2 0 1 3 3 1 6 2 4 6 2 6 2 2 2 2 5 2 3 3 4 0 6 3 0 2 6 3 1 6 6 6 3 4 3 3 3 4 6 3 1 1 3 3 2 2 2 2 6 1 1 2 6 2 2 6 4 2 2 6 5 2 2 2 0 6 1 1 4 4 1 1 0 1 6 1 5 4 6 1 3 3 1 3 2 0 2 0 6 6 1 0 6 6 3 0 2 2 1 2 2 4 3], shape=(128,), dtype=int64)
Model definition
In order to demonstrate the use of graph regularization, we build a base model
for this problem first. We will use a simple feed-forward neural network with 2
hidden layers and dropout in between. We illustrate the creation of the base
model using all model types supported by the tf.Keras framework -- sequential,
functional, and subclass.
Sequential base model
def make_mlp_sequential_model(hparams):
"""Creates a sequential multi-layer perceptron model."""
model = tf.keras.Sequential()
model.add(
tf.keras.layers.InputLayer(
input_shape=(hparams.max_seq_length,), name='words'))
# Input is already one-hot encoded in the integer format. We cast it to
# floating point format here.
model.add(
tf.keras.layers.Lambda(lambda x: tf.keras.backend.cast(x, tf.float32)))
for num_units in hparams.num_fc_units:
model.add(tf.keras.layers.Dense(num_units, activation='relu'))
# For sequential models, by default, Keras ensures that the 'dropout' layer
# is invoked only during training.
model.add(tf.keras.layers.Dropout(hparams.dropout_rate))
model.add(tf.keras.layers.Dense(hparams.num_classes, activation='softmax'))
return model
Functional base model
def make_mlp_functional_model(hparams):
"""Creates a functional API-based multi-layer perceptron model."""
inputs = tf.keras.Input(
shape=(hparams.max_seq_length,), dtype='int64', name='words')
# Input is already one-hot encoded in the integer format. We cast it to
# floating point format here.
cur_layer = tf.keras.layers.Lambda(
lambda x: tf.keras.backend.cast(x, tf.float32))(
inputs)
for num_units in hparams.num_fc_units:
cur_layer = tf.keras.layers.Dense(num_units, activation='relu')(cur_layer)
# For functional models, by default, Keras ensures that the 'dropout' layer
# is invoked only during training.
cur_layer = tf.keras.layers.Dropout(hparams.dropout_rate)(cur_layer)
outputs = tf.keras.layers.Dense(
hparams.num_classes, activation='softmax')(
cur_layer)
model = tf.keras.Model(inputs, outputs=outputs)
return model
Subclass base model
def make_mlp_subclass_model(hparams):
"""Creates a multi-layer perceptron subclass model in Keras."""
class MLP(tf.keras.Model):
"""Subclass model defining a multi-layer perceptron."""
def __init__(self):
super(MLP, self).__init__()
# Input is already one-hot encoded in the integer format. We create a
# layer to cast it to floating point format here.
self.cast_to_float_layer = tf.keras.layers.Lambda(
lambda x: tf.keras.backend.cast(x, tf.float32))
self.dense_layers = [
tf.keras.layers.Dense(num_units, activation='relu')
for num_units in hparams.num_fc_units
]
self.dropout_layer = tf.keras.layers.Dropout(hparams.dropout_rate)
self.output_layer = tf.keras.layers.Dense(
hparams.num_classes, activation='softmax')
def call(self, inputs, training=False):
cur_layer = self.cast_to_float_layer(inputs['words'])
for dense_layer in self.dense_layers:
cur_layer = dense_layer(cur_layer)
cur_layer = self.dropout_layer(cur_layer, training=training)
outputs = self.output_layer(cur_layer)
return outputs
return MLP()
Create base model(s)
# Create a base MLP model using the functional API.
# Alternatively, you can also create a sequential or subclass base model using
# the make_mlp_sequential_model() or make_mlp_subclass_model() functions
# respectively, defined above. Note that if a subclass model is used, its
# summary cannot be generated until it is built.
base_model_tag, base_model = 'FUNCTIONAL', make_mlp_functional_model(HPARAMS)
base_model.summary()
Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= words (InputLayer) [(None, 1433)] 0 _________________________________________________________________ lambda (Lambda) (None, 1433) 0 _________________________________________________________________ dense (Dense) (None, 50) 71700 _________________________________________________________________ dropout (Dropout) (None, 50) 0 _________________________________________________________________ dense_1 (Dense) (None, 50) 2550 _________________________________________________________________ dropout_1 (Dropout) (None, 50) 0 _________________________________________________________________ dense_2 (Dense) (None, 7) 357 ================================================================= Total params: 74,607 Trainable params: 74,607 Non-trainable params: 0 _________________________________________________________________
Train base MLP model
# Compile and train the base MLP model
base_model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
base_model.fit(train_dataset, epochs=HPARAMS.train_epochs, verbose=1)
Epoch 1/100 17/17 [==============================] - 1s 36ms/step - loss: 1.9138 - accuracy: 0.2088 Epoch 2/100 17/17 [==============================] - 0s 16ms/step - loss: 1.8343 - accuracy: 0.3007 Epoch 3/100 17/17 [==============================] - 0s 16ms/step - loss: 1.7315 - accuracy: 0.3434 Epoch 4/100 17/17 [==============================] - 0s 16ms/step - loss: 1.6118 - accuracy: 0.3898 Epoch 5/100 17/17 [==============================] - 0s 16ms/step - loss: 1.4899 - accuracy: 0.4515 Epoch 6/100 17/17 [==============================] - 0s 16ms/step - loss: 1.3573 - accuracy: 0.5114 Epoch 7/100 17/17 [==============================] - 0s 16ms/step - loss: 1.2113 - accuracy: 0.5907 Epoch 8/100 17/17 [==============================] - 0s 16ms/step - loss: 1.0746 - accuracy: 0.6399 Epoch 9/100 17/17 [==============================] - 0s 15ms/step - loss: 0.9746 - accuracy: 0.6729 Epoch 10/100 17/17 [==============================] - 0s 16ms/step - loss: 0.9033 - accuracy: 0.6956 Epoch 11/100 17/17 [==============================] - 0s 15ms/step - loss: 0.7879 - accuracy: 0.7360 Epoch 12/100 17/17 [==============================] - 0s 15ms/step - loss: 0.7073 - accuracy: 0.7652 Epoch 13/100 17/17 [==============================] - 0s 16ms/step - loss: 0.6644 - accuracy: 0.7782 Epoch 14/100 17/17 [==============================] - 0s 15ms/step - loss: 0.6027 - accuracy: 0.8116 Epoch 15/100 17/17 [==============================] - 0s 15ms/step - loss: 0.5522 - accuracy: 0.8278 Epoch 16/100 17/17 [==============================] - 0s 15ms/step - loss: 0.5034 - accuracy: 0.8413 Epoch 17/100 17/17 [==============================] - 0s 15ms/step - loss: 0.4774 - accuracy: 0.8492 Epoch 18/100 17/17 [==============================] - 0s 15ms/step - loss: 0.4477 - accuracy: 0.8524 Epoch 19/100 17/17 [==============================] - 0s 15ms/step - loss: 0.4221 - accuracy: 0.8752 Epoch 20/100 17/17 [==============================] - 0s 14ms/step - loss: 0.3878 - accuracy: 0.8803 Epoch 21/100 17/17 [==============================] - 0s 15ms/step - loss: 0.3373 - accuracy: 0.9077 Epoch 22/100 17/17 [==============================] - 0s 15ms/step - loss: 0.3338 - accuracy: 0.8993 Epoch 23/100 17/17 [==============================] - 0s 15ms/step - loss: 0.3197 - accuracy: 0.9030 Epoch 24/100 17/17 [==============================] - 0s 15ms/step - loss: 0.3329 - accuracy: 0.9012 Epoch 25/100 17/17 [==============================] - 0s 15ms/step - loss: 0.2936 - accuracy: 0.9118 Epoch 26/100 17/17 [==============================] - 0s 15ms/step - loss: 0.2667 - accuracy: 0.9220 Epoch 27/100 17/17 [==============================] - 0s 14ms/step - loss: 0.2549 - accuracy: 0.9253 Epoch 28/100 17/17 [==============================] - 0s 15ms/step - loss: 0.2588 - accuracy: 0.9230 Epoch 29/100 17/17 [==============================] - 0s 15ms/step - loss: 0.2217 - accuracy: 0.9369 Epoch 30/100 17/17 [==============================] - 0s 15ms/step - loss: 0.2369 - accuracy: 0.9271 Epoch 31/100 17/17 [==============================] - 0s 15ms/step - loss: 0.2057 - accuracy: 0.9355 Epoch 32/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1893 - accuracy: 0.9457 Epoch 33/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1834 - accuracy: 0.9480 Epoch 34/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1945 - accuracy: 0.9439 Epoch 35/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1788 - accuracy: 0.9466 Epoch 36/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1712 - accuracy: 0.9550 Epoch 37/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1701 - accuracy: 0.9513 Epoch 38/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1646 - accuracy: 0.9448 Epoch 39/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1546 - accuracy: 0.9513 Epoch 40/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1638 - accuracy: 0.9490 Epoch 41/100 17/17 [==============================] - 0s 14ms/step - loss: 0.1493 - accuracy: 0.9582 Epoch 42/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1511 - accuracy: 0.9582 Epoch 43/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1428 - accuracy: 0.9619 Epoch 44/100 17/17 [==============================] - 0s 14ms/step - loss: 0.1315 - accuracy: 0.9643 Epoch 45/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1348 - accuracy: 0.9587 Epoch 46/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1216 - accuracy: 0.9657 Epoch 47/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1184 - accuracy: 0.9675 Epoch 48/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1071 - accuracy: 0.9726 Epoch 49/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1081 - accuracy: 0.9698 Epoch 50/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1045 - accuracy: 0.9684 Epoch 51/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0928 - accuracy: 0.9763 Epoch 52/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1212 - accuracy: 0.9638 Epoch 53/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1043 - accuracy: 0.9717 Epoch 54/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0966 - accuracy: 0.9745 Epoch 55/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0906 - accuracy: 0.9726 Epoch 56/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1012 - accuracy: 0.9694 Epoch 57/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0837 - accuracy: 0.9749 Epoch 58/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0840 - accuracy: 0.9768 Epoch 59/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1008 - accuracy: 0.9731 Epoch 60/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0869 - accuracy: 0.9745 Epoch 61/100 17/17 [==============================] - 0s 14ms/step - loss: 0.0820 - accuracy: 0.9763 Epoch 62/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0813 - accuracy: 0.9782 Epoch 63/100 17/17 [==============================] - 0s 14ms/step - loss: 0.0876 - accuracy: 0.9735 Epoch 64/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0832 - accuracy: 0.9768 Epoch 65/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0759 - accuracy: 0.9768 Epoch 66/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0758 - accuracy: 0.9791 Epoch 67/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0880 - accuracy: 0.9726 Epoch 68/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0760 - accuracy: 0.9777 Epoch 69/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0659 - accuracy: 0.9842 Epoch 70/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0669 - accuracy: 0.9814 Epoch 71/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0699 - accuracy: 0.9810 Epoch 72/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0682 - accuracy: 0.9800 Epoch 73/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0654 - accuracy: 0.9833 Epoch 74/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0667 - accuracy: 0.9824 Epoch 75/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0663 - accuracy: 0.9819 Epoch 76/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0629 - accuracy: 0.9819 Epoch 77/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0714 - accuracy: 0.9773 Epoch 78/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0664 - accuracy: 0.9800 Epoch 79/100 17/17 [==============================] - 0s 14ms/step - loss: 0.0639 - accuracy: 0.9814 Epoch 80/100 17/17 [==============================] - 0s 14ms/step - loss: 0.0597 - accuracy: 0.9833 Epoch 81/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0533 - accuracy: 0.9861 Epoch 82/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0575 - accuracy: 0.9838 Epoch 83/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0540 - accuracy: 0.9847 Epoch 84/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0550 - accuracy: 0.9847 Epoch 85/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0664 - accuracy: 0.9824 Epoch 86/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0533 - accuracy: 0.9884 Epoch 87/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0540 - accuracy: 0.9856 Epoch 88/100 17/17 [==============================] - 0s 14ms/step - loss: 0.0583 - accuracy: 0.9842 Epoch 89/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0508 - accuracy: 0.9842 Epoch 90/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0640 - accuracy: 0.9787 Epoch 91/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0500 - accuracy: 0.9861 Epoch 92/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0623 - accuracy: 0.9828 Epoch 93/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0480 - accuracy: 0.9861 Epoch 94/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0503 - accuracy: 0.9879 Epoch 95/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0548 - accuracy: 0.9833 Epoch 96/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0471 - accuracy: 0.9875 Epoch 97/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0515 - accuracy: 0.9865 Epoch 98/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0435 - accuracy: 0.9912 Epoch 99/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0451 - accuracy: 0.9893 Epoch 100/100 17/17 [==============================] - 0s 15ms/step - loss: 0.0530 - accuracy: 0.9824 <tensorflow.python.keras.callbacks.History at 0x7fc46428e208>
Evaluate base MLP model
# Helper function to print evaluation metrics.
def print_metrics(model_desc, eval_metrics):
"""Prints evaluation metrics.
Args:
model_desc: A description of the model.
eval_metrics: A dictionary mapping metric names to corresponding values. It
must contain the loss and accuracy metrics.
"""
print('\n')
print('Eval accuracy for ', model_desc, ': ', eval_metrics['accuracy'])
print('Eval loss for ', model_desc, ': ', eval_metrics['loss'])
if 'graph_loss' in eval_metrics:
print('Eval graph loss for ', model_desc, ': ', eval_metrics['graph_loss'])
eval_results = dict(
zip(base_model.metrics_names,
base_model.evaluate(test_dataset, steps=HPARAMS.eval_steps)))
print_metrics('Base MLP model', eval_results)
5/Unknown - 0s 26ms/step - loss: 1.4019 - accuracy: 0.7993
Eval accuracy for Base MLP model : 0.79927665
Eval loss for Base MLP model : 1.401930844783783
Train MLP model with graph regularization
Incorporating graph regularization into the loss term of an existing
tf.Keras.Model requires just a few lines of code. The base model is wrapped to
create a new tf.Keras subclass model, whose loss includes graph
regularization.
To assess the incremental benefit of graph regularization, we will create a new
base model instance. This is because base_model has already been trained for a
few iterations, and reusing this trained model to create a graph-regularized
model will not be a fair comparison for base_model.
# Build a new base MLP model.
base_reg_model_tag, base_reg_model = 'FUNCTIONAL', make_mlp_functional_model(
HPARAMS)
# Wrap the base MLP model with graph regularization.
graph_reg_config = nsl.configs.make_graph_reg_config(
max_neighbors=HPARAMS.num_neighbors,
multiplier=HPARAMS.graph_regularization_multiplier,
distance_type=HPARAMS.distance_type,
sum_over_axis=-1)
graph_reg_model = nsl.keras.GraphRegularization(base_reg_model,
graph_reg_config)
graph_reg_model.compile(
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
graph_reg_model.fit(train_dataset, epochs=HPARAMS.train_epochs, verbose=1)
Epoch 1/100 /tmpfs/src/tf_docs_env/lib/python3.5/site-packages/tensorflow_core/python/framework/indexed_slices.py:434: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " 17/17 [==============================] - 2s 97ms/step - loss: 1.9373 - accuracy: 0.1865 - graph_loss: 0.0075 Epoch 2/100 17/17 [==============================] - 0s 17ms/step - loss: 1.8416 - accuracy: 0.3026 - graph_loss: 0.0112 Epoch 3/100 17/17 [==============================] - 0s 16ms/step - loss: 1.7563 - accuracy: 0.3253 - graph_loss: 0.0263 Epoch 4/100 17/17 [==============================] - 0s 18ms/step - loss: 1.6447 - accuracy: 0.3749 - graph_loss: 0.0453 Epoch 5/100 17/17 [==============================] - 0s 19ms/step - loss: 1.5385 - accuracy: 0.4209 - graph_loss: 0.0667 Epoch 6/100 17/17 [==============================] - 0s 17ms/step - loss: 1.3956 - accuracy: 0.4961 - graph_loss: 0.1006 Epoch 7/100 17/17 [==============================] - 0s 17ms/step - loss: 1.2717 - accuracy: 0.5745 - graph_loss: 0.1349 Epoch 8/100 17/17 [==============================] - 0s 17ms/step - loss: 1.1315 - accuracy: 0.6445 - graph_loss: 0.1644 Epoch 9/100 17/17 [==============================] - 0s 18ms/step - loss: 1.0278 - accuracy: 0.6742 - graph_loss: 0.2103 Epoch 10/100 17/17 [==============================] - 0s 17ms/step - loss: 0.9069 - accuracy: 0.7123 - graph_loss: 0.2406 Epoch 11/100 17/17 [==============================] - 0s 17ms/step - loss: 0.8342 - accuracy: 0.7378 - graph_loss: 0.2588 Epoch 12/100 17/17 [==============================] - 0s 18ms/step - loss: 0.7628 - accuracy: 0.7684 - graph_loss: 0.2693 Epoch 13/100 17/17 [==============================] - 0s 18ms/step - loss: 0.6870 - accuracy: 0.7972 - graph_loss: 0.2766 Epoch 14/100 17/17 [==============================] - 0s 18ms/step - loss: 0.6566 - accuracy: 0.8074 - graph_loss: 0.2972 Epoch 15/100 17/17 [==============================] - 0s 18ms/step - loss: 0.5953 - accuracy: 0.8320 - graph_loss: 0.3061 Epoch 16/100 17/17 [==============================] - 0s 17ms/step - loss: 0.5459 - accuracy: 0.8445 - graph_loss: 0.3125 Epoch 17/100 17/17 [==============================] - 0s 17ms/step - loss: 0.5279 - accuracy: 0.8501 - graph_loss: 0.3062 Epoch 18/100 17/17 [==============================] - 0s 17ms/step - loss: 0.4872 - accuracy: 0.8645 - graph_loss: 0.3185 Epoch 19/100 17/17 [==============================] - 0s 18ms/step - loss: 0.4523 - accuracy: 0.8821 - graph_loss: 0.3077 Epoch 20/100 17/17 [==============================] - 0s 19ms/step - loss: 0.4131 - accuracy: 0.8849 - graph_loss: 0.3145 Epoch 21/100 17/17 [==============================] - 0s 19ms/step - loss: 0.3878 - accuracy: 0.9007 - graph_loss: 0.3190 Epoch 22/100 17/17 [==============================] - 0s 19ms/step - loss: 0.3564 - accuracy: 0.9053 - graph_loss: 0.3211 Epoch 23/100 17/17 [==============================] - 0s 18ms/step - loss: 0.3448 - accuracy: 0.9132 - graph_loss: 0.3296 Epoch 24/100 17/17 [==============================] - 0s 19ms/step - loss: 0.3292 - accuracy: 0.9123 - graph_loss: 0.3297 Epoch 25/100 17/17 [==============================] - 0s 18ms/step - loss: 0.3060 - accuracy: 0.9248 - graph_loss: 0.3372 Epoch 26/100 17/17 [==============================] - 0s 18ms/step - loss: 0.2889 - accuracy: 0.9285 - graph_loss: 0.3407 Epoch 27/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2838 - accuracy: 0.9309 - graph_loss: 0.3327 Epoch 28/100 17/17 [==============================] - 0s 18ms/step - loss: 0.2800 - accuracy: 0.9336 - graph_loss: 0.3334 Epoch 29/100 17/17 [==============================] - 0s 18ms/step - loss: 0.2650 - accuracy: 0.9327 - graph_loss: 0.3404 Epoch 30/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2627 - accuracy: 0.9350 - graph_loss: 0.3528 Epoch 31/100 17/17 [==============================] - 0s 18ms/step - loss: 0.2308 - accuracy: 0.9490 - graph_loss: 0.3339 Epoch 32/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2316 - accuracy: 0.9466 - graph_loss: 0.3341 Epoch 33/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2138 - accuracy: 0.9522 - graph_loss: 0.3339 Epoch 34/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2074 - accuracy: 0.9564 - graph_loss: 0.3391 Epoch 35/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2185 - accuracy: 0.9508 - graph_loss: 0.3375 Epoch 36/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2073 - accuracy: 0.9531 - graph_loss: 0.3514 Epoch 37/100 17/17 [==============================] - 0s 17ms/step - loss: 0.2047 - accuracy: 0.9531 - graph_loss: 0.3569 Epoch 38/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1758 - accuracy: 0.9643 - graph_loss: 0.3521 Epoch 39/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1832 - accuracy: 0.9564 - graph_loss: 0.3526 Epoch 40/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1846 - accuracy: 0.9587 - graph_loss: 0.3412 Epoch 41/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1731 - accuracy: 0.9657 - graph_loss: 0.3424 Epoch 42/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1654 - accuracy: 0.9698 - graph_loss: 0.3603 Epoch 43/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1624 - accuracy: 0.9698 - graph_loss: 0.3356 Epoch 44/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1602 - accuracy: 0.9629 - graph_loss: 0.3383 Epoch 45/100 17/17 [==============================] - 0s 18ms/step - loss: 0.1563 - accuracy: 0.9657 - graph_loss: 0.3529 Epoch 46/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1440 - accuracy: 0.9740 - graph_loss: 0.3388 Epoch 47/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1426 - accuracy: 0.9745 - graph_loss: 0.3551 Epoch 48/100 17/17 [==============================] - 0s 18ms/step - loss: 0.1425 - accuracy: 0.9722 - graph_loss: 0.3553 Epoch 49/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1475 - accuracy: 0.9680 - graph_loss: 0.3411 Epoch 50/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1368 - accuracy: 0.9717 - graph_loss: 0.3494 Epoch 51/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1245 - accuracy: 0.9763 - graph_loss: 0.3320 Epoch 52/100 17/17 [==============================] - 0s 15ms/step - loss: 0.1286 - accuracy: 0.9754 - graph_loss: 0.3495 Epoch 53/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1336 - accuracy: 0.9759 - graph_loss: 0.3387 Epoch 54/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1285 - accuracy: 0.9773 - graph_loss: 0.3475 Epoch 55/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1154 - accuracy: 0.9810 - graph_loss: 0.3375 Epoch 56/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1337 - accuracy: 0.9703 - graph_loss: 0.3471 Epoch 57/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1165 - accuracy: 0.9796 - graph_loss: 0.3426 Epoch 58/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1395 - accuracy: 0.9712 - graph_loss: 0.3452 Epoch 59/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1107 - accuracy: 0.9800 - graph_loss: 0.3350 Epoch 60/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1154 - accuracy: 0.9763 - graph_loss: 0.3410 Epoch 61/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1165 - accuracy: 0.9791 - graph_loss: 0.3485 Epoch 62/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1185 - accuracy: 0.9759 - graph_loss: 0.3453 Epoch 63/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1191 - accuracy: 0.9791 - graph_loss: 0.3367 Epoch 64/100 17/17 [==============================] - 0s 16ms/step - loss: 0.1102 - accuracy: 0.9800 - graph_loss: 0.3429 Epoch 65/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1106 - accuracy: 0.9782 - graph_loss: 0.3414 Epoch 66/100 17/17 [==============================] - 0s 18ms/step - loss: 0.1072 - accuracy: 0.9800 - graph_loss: 0.3373 Epoch 67/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1016 - accuracy: 0.9796 - graph_loss: 0.3405 Epoch 68/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1056 - accuracy: 0.9800 - graph_loss: 0.3466 Epoch 69/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0931 - accuracy: 0.9865 - graph_loss: 0.3447 Epoch 70/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1000 - accuracy: 0.9842 - graph_loss: 0.3372 Epoch 71/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0972 - accuracy: 0.9838 - graph_loss: 0.3418 Epoch 72/100 17/17 [==============================] - 0s 17ms/step - loss: 0.1026 - accuracy: 0.9847 - graph_loss: 0.3513 Epoch 73/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0987 - accuracy: 0.9814 - graph_loss: 0.3432 Epoch 74/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0948 - accuracy: 0.9819 - graph_loss: 0.3419 Epoch 75/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0953 - accuracy: 0.9819 - graph_loss: 0.3419 Epoch 76/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0972 - accuracy: 0.9833 - graph_loss: 0.3394 Epoch 77/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0977 - accuracy: 0.9828 - graph_loss: 0.3513 Epoch 78/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0911 - accuracy: 0.9847 - graph_loss: 0.3482 Epoch 79/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0873 - accuracy: 0.9861 - graph_loss: 0.3372 Epoch 80/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0994 - accuracy: 0.9819 - graph_loss: 0.3415 Epoch 81/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0938 - accuracy: 0.9852 - graph_loss: 0.3484 Epoch 82/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0891 - accuracy: 0.9879 - graph_loss: 0.3378 Epoch 83/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0883 - accuracy: 0.9861 - graph_loss: 0.3480 Epoch 84/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0923 - accuracy: 0.9833 - graph_loss: 0.3469 Epoch 85/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0837 - accuracy: 0.9893 - graph_loss: 0.3446 Epoch 86/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0862 - accuracy: 0.9856 - graph_loss: 0.3500 Epoch 87/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0836 - accuracy: 0.9875 - graph_loss: 0.3422 Epoch 88/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0846 - accuracy: 0.9879 - graph_loss: 0.3435 Epoch 89/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0844 - accuracy: 0.9856 - graph_loss: 0.3289 Epoch 90/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0820 - accuracy: 0.9865 - graph_loss: 0.3422 Epoch 91/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0788 - accuracy: 0.9893 - graph_loss: 0.3316 Epoch 92/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0878 - accuracy: 0.9847 - graph_loss: 0.3388 Epoch 93/100 17/17 [==============================] - 0s 18ms/step - loss: 0.0827 - accuracy: 0.9870 - graph_loss: 0.3395 Epoch 94/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0731 - accuracy: 0.9898 - graph_loss: 0.3401 Epoch 95/100 17/17 [==============================] - 0s 16ms/step - loss: 0.0864 - accuracy: 0.9852 - graph_loss: 0.3337 Epoch 96/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0846 - accuracy: 0.9861 - graph_loss: 0.3432 Epoch 97/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0785 - accuracy: 0.9889 - graph_loss: 0.3360 Epoch 98/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0719 - accuracy: 0.9921 - graph_loss: 0.3399 Epoch 99/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0793 - accuracy: 0.9879 - graph_loss: 0.3476 Epoch 100/100 17/17 [==============================] - 0s 17ms/step - loss: 0.0763 - accuracy: 0.9884 - graph_loss: 0.3444 <tensorflow.python.keras.callbacks.History at 0x7fc428f54550>
Evaluate MLP model with graph regularization
eval_results = dict(
zip(graph_reg_model.metrics_names,
graph_reg_model.evaluate(test_dataset, steps=HPARAMS.eval_steps)))
print_metrics('MLP + graph regularization', eval_results)
5/Unknown - 0s 83ms/step - loss: 1.2183 - accuracy: 0.7957 - graph_loss: 0.0000e+00
Eval accuracy for MLP + graph regularization : 0.79566
Eval loss for MLP + graph regularization : 1.2183328151702881
Eval graph loss for MLP + graph regularization : 0.0
The graph-regularized model's accuracy is about 2-3% higher than that of the
base model (base_model).
Conclusion
We have demonstrated the use of graph regularization for document classification on a natural citation graph (Cora) using the Neural Structured Learning (NSL) framework. Our advanced tutorial involves synthesizing graphs based on sample embeddings before training a neural network with graph regularization. This approach is useful if the input does not contain an explicit graph.
We encourage users to experiment further by varying the amount of supervision as well as trying different neural architectures for graph regularization.