TF 2.0 is out! Get hands-on practice at TF World, Oct 28-31. Use code TF20 for 20% off select passes. Register now

TensorFlow Model Analysis

An Example of a Key Component of TensorFlow Extended (TFX)

This example colab notebook illustrates how TensorFlow Model Analysis (TFMA) can be used to investigate and visualize the characteristics of a dataset and the performance of a model. We'll use a model that we trained previously, and now you get to play with the results!

The model we trained was for the Chicago Taxi Example, which uses the Taxi Trips dataset released by the City of Chicago.

Read more about the dataset in Google BigQuery. Explore the full dataset in the BigQuery UI.

The columns in the dataset are:

pickup_community_areafaretrip_start_month
trip_start_hourtrip_start_daytrip_start_timestamp
pickup_latitudepickup_longitudedropoff_latitude
dropoff_longitudetrip_milespickup_census_tract
dropoff_census_tractpayment_typecompany
trip_secondsdropoff_community_areatips

Install Jupyter Extensions

jupyter nbextension enable --py widgetsnbextension
jupyter nbextension install --py --symlink tensorflow_model_analysis
jupyter nbextension enable --py tensorflow_model_analysis

Install TensorFlow Model Analysis (TFMA)

This will pull in all the dependencies, and will take a minute. Please ignore the warnings.

import sys, os
# Confirm that we're using Python 3
assert sys.version_info.major is 3, 'Oops, not running Python 3'

# Install TFMA
# This will pull in all the dependencies, and will take a minute
# Please ignore the warnings
!pip install -q tensorflow
!pip install -q tensorflow_model_analysis=='0.14.0'

import tensorflow as tf
import tensorflow_model_analysis as tfma
tf.logging.set_verbosity(tf.logging.ERROR)
print('TFMA version: {}'.format(tfma.version.VERSION_STRING))
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/apache_beam/__init__.py:84: UserWarning: Some syntactic constructs of Python 3 are not yet fully supported by Apache Beam.
  'Some syntactic constructs of Python 3 are not yet fully supported by '

TFMA version: 0.14.0

Load The Files

We'll download a zip file that has everything we need. That includes:

  • Training and evaluation datasets
  • Data schema
  • Training results as EvalSavedModels
# Download the zip file from GCP and unzip it
import tempfile, requests, zipfile, io
BASE_DIR = tempfile.mkdtemp()
TFMA_DIR = os.path.join(BASE_DIR, 'eval_saved_models-0.11.0')
DATA_DIR = os.path.join(TFMA_DIR, 'data')
OUTPUT_DIR = os.path.join(TFMA_DIR, 'output')
SCHEMA = os.path.join(TFMA_DIR, 'schema.pbtxt')

response = requests.get('https://storage.googleapis.com/tfx-colab-datasets/eval_saved_models-0.11.0.zip', stream=True)
zipfile.ZipFile(io.BytesIO(response.content)).extractall(BASE_DIR)

print("Here's what we downloaded:")
!ls -R {TFMA_DIR}
Here's what we downloaded:
/tmp/tmptjs_8kc1/eval_saved_models-0.11.0:
data  run_0  run_1  run_2  schema.pbtxt

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/data:
eval  train

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/data/eval:
data.csv

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/data/train:
data.csv

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_0:
eval_model_dir

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_0/eval_model_dir:
1544225322

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_0/eval_model_dir/1544225322:
assets  saved_model.pb  variables

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_0/eval_model_dir/1544225322/assets:
vocab_compute_and_apply_vocabulary_1_vocabulary
vocab_compute_and_apply_vocabulary_vocabulary

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_0/eval_model_dir/1544225322/variables:
variables.data-00000-of-00002  variables.data-00001-of-00002  variables.index

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_1:
eval_model_dir

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_1/eval_model_dir:
1544225718

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_1/eval_model_dir/1544225718:
assets  saved_model.pb  variables

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_1/eval_model_dir/1544225718/assets:
vocab_compute_and_apply_vocabulary_1_vocabulary
vocab_compute_and_apply_vocabulary_vocabulary

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_1/eval_model_dir/1544225718/variables:
variables.data-00000-of-00002  variables.data-00001-of-00002  variables.index

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_2:
eval_model_dir

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_2/eval_model_dir:
1544225766

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_2/eval_model_dir/1544225766:
assets  saved_model.pb  variables

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_2/eval_model_dir/1544225766/assets:
vocab_compute_and_apply_vocabulary_1_vocabulary
vocab_compute_and_apply_vocabulary_vocabulary

/tmp/tmptjs_8kc1/eval_saved_models-0.11.0/run_2/eval_model_dir/1544225766/variables:
variables.data-00000-of-00002  variables.data-00001-of-00002  variables.index

Parse the Schema

Among the things we downloaded was a schema for our data that was created by TensorFlow Data Validation. Let's parse that now so that we can use it with TFMA.

from google.protobuf import text_format
from tensorflow.python.lib.io import file_io
from tensorflow_metadata.proto.v0 import schema_pb2
from tensorflow.core.example import example_pb2
from tensorflow import python_io

schema = schema_pb2.Schema()
contents = file_io.read_file_to_string(SCHEMA)
schema = text_format.Parse(contents, schema)

Use the Schema to Create TFRecords

We need to give TFMA access to our dataset, so let's create a TFRecords file. We can use our schema to create it, since it gives us the correct type for each feature.

import csv

datafile = os.path.join(DATA_DIR, 'eval', 'data.csv')
reader = csv.DictReader(open(datafile, 'r'))
examples = []
for line in reader:
  example = example_pb2.Example()
  for feature in schema.feature:
    key = feature.name
    if len(line[key]) > 0:
      if feature.type == schema_pb2.FLOAT:
        example.features.feature[key].float_list.value[:] = [float(line[key])]
      elif feature.type == schema_pb2.INT:
        example.features.feature[key].int64_list.value[:] = [int(line[key])]
      elif feature.type == schema_pb2.BYTES:
        example.features.feature[key].bytes_list.value[:] = [line[key].encode('utf8')]
    else:
      if feature.type == schema_pb2.FLOAT:
        example.features.feature[key].float_list.value[:] = []
      elif feature.type == schema_pb2.INT:
        example.features.feature[key].int64_list.value[:] = []
      elif feature.type == schema_pb2.BYTES:
        example.features.feature[key].bytes_list.value[:] = []
  examples.append(example)

TFRecord_file = os.path.join(BASE_DIR, 'train_data.rio')
with python_io.TFRecordWriter(TFRecord_file) as writer:
  for example in examples:
    writer.write(example.SerializeToString())
  writer.flush()
  writer.close()

!ls {TFRecord_file}
/tmp/tmptjs_8kc1/train_data.rio

Run TFMA and Render Metrics

Now we're ready to create a function that we'll use to run TFMA and render metrics. It requires an EvalSavedModel, a list of SliceSpecs, and an index into the SliceSpec list. It will create an EvalResult using tfma.run_model_analysis, and use it to create a SlicingMetricsViewer using tfma.view.render_slicing_metrics, which will render a visualization of our dataset using the slice we created.

def run_and_render(eval_model=None, slice_list=None, slice_idx=0):
  """Runs the model analysis and renders the slicing metrics

  Args:
      eval_model: An instance of tf.saved_model saved with evaluation data
      slice_list: A list of tfma.slicer.SingleSliceSpec giving the slices
      slice_idx: An integer index into slice_list specifying the slice to use

  Returns:
      A SlicingMetricsViewer object if in Jupyter notebook; None if in Colab.
  """
  eval_result = tfma.run_model_analysis(eval_shared_model=eval_model,
                                          data_location=TFRecord_file,
                                          file_format='tfrecords',
                                          slice_spec=slice_list,
                                          output_path='sample_data',
                                          extractors=None)
  return tfma.view.render_slicing_metrics(eval_result, slicing_spec=slices[slice_idx])

Slicing and Dicing

We previously trained a model, and now we've loaded the results. Let's take a look at our visualizations, starting with using TFMA to slice along particular features. But first we need to read in the EvalSavedModel from one of our previous training runs.

Plots are interactive:

  • Click and drag to pan
  • Scroll to zoom
  • Right click to reset the view

Simply hover over the desired data point to see more details. Select from four different types of plots using the selections at the bottom.

For example, we'll be setting slicing_column to look at the trip_start_hour feature in our SliceSpec.

# Load the TFMA results for the first training run
# This will take a minute
eval_model_base_dir_0 = os.path.join(TFMA_DIR, 'run_0', 'eval_model_dir')
eval_model_dir_0 = os.path.join(eval_model_base_dir_0, next(os.walk(eval_model_base_dir_0))[1][0])
eval_shared_model_0 = tfma.default_eval_shared_model(eval_saved_model_path=eval_model_dir_0)

# Slice our data by the trip_start_hour feature
slices = [tfma.slicer.SingleSliceSpec(columns=['trip_start_hour'])]

run_and_render(eval_model=eval_shared_model_0, slice_list=slices, slice_idx=0)
WARNING:root:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.

SlicingMetricsViewer(config={'weightedExamplesColumn': 'post_export_metrics/example_count'}, data=[{'slice': '…

Slices Overview

The default visualization is the Slices Overview when the number of slices is small. It shows the values of metrics for each slice. Since we've selected trip_start_hour above, it's showing us metrics like accuracy and AUC for each hour, which allows us to look for issues that are specific to some hours and not others.

In the visualization above:

  • Try sorting the feature column, which is our trip_start_hours feature, by clicking on the column header
  • Try sorting by precision, and notice that the precision for some of the hours with examples is 0, which may indicate a problem

The chart also allows us to select and display different metrics in our slices.

  • Try selecting different metrics from the "Show" menu
  • Try selecting recall in the "Show" menu, and notice that the recall for some of the hours with examples is 0, which may indicate a problem

It is also possible to set a threshold to filter out slices with smaller numbers of examples, or "weights". You can type a minimum number of examples, or use the slider.

Metrics Histogram

This view also supports a Metrics Histogram as an alternative visualization, which is also the default view when the number of slices is large. The results will be divided into buckets and the number of slices / total weights / both can be visualized. Columns can be sorted by clicking on the column header. Slices with small weights can be filtered out by setting the threshold. Further filtering can be applied by dragging the grey band. To reset the range, double click the band. Filtering can also be used to remove outliers in the visualization and the metrics tables. Click the gear icon to switch to a logarithmic scale instead of a linear scale.

  • Try selecting "Metrics Histogram" in the Visualization menu

More Slices

Let's create a whole list of SliceSpecs, which will allow us to select any of the slices in the list. We'll select the trip_start_day slice (days of the week) by setting the slice_idx to 1. Try changing the slice_idx to 0 or 2 and running again to examine different slices.

slices = [tfma.slicer.SingleSliceSpec(columns=['trip_start_hour']),
          tfma.slicer.SingleSliceSpec(columns=['trip_start_day']),
          tfma.slicer.SingleSliceSpec(columns=['trip_start_month'])]
run_and_render(eval_model=eval_shared_model_0, slice_list=slices, slice_idx=1)
WARNING:root:Deleting 1 existing files in target path matching: 
WARNING:root:Deleting 1 existing files in target path matching: 
WARNING:root:Deleting 1 existing files in target path matching: 

SlicingMetricsViewer(config={'weightedExamplesColumn': 'post_export_metrics/example_count'}, data=[{'slice': '…

You can create feature crosses to analyze combinations of features. Let's create a SliceSpec to look at a cross of trip_start_day and trip_start_hour:

slices = [tfma.slicer.SingleSliceSpec(columns=['trip_start_day', 'trip_start_hour'])]
run_and_render(eval_shared_model_0, slices, 0)
WARNING:root:Deleting 1 existing files in target path matching: 
WARNING:root:Deleting 1 existing files in target path matching: 
WARNING:root:Deleting 1 existing files in target path matching: 

SlicingMetricsViewer(config={'weightedExamplesColumn': 'post_export_metrics/example_count'}, data=[{'slice': '…

Crossing the two columns creates a lot of combinations! Let's narrow down our cross to only look at trips that start at noon. Then let's select accuracy from the visualization:

slices = [tfma.slicer.SingleSliceSpec(columns=['trip_start_day'], features=[('trip_start_hour', 12)])]
run_and_render(eval_shared_model_0, slices, 0)
WARNING:root:Deleting 1 existing files in target path matching: 
WARNING:root:Deleting 1 existing files in target path matching: 
WARNING:root:Deleting 1 existing files in target path matching: 

SlicingMetricsViewer(config={'weightedExamplesColumn': 'post_export_metrics/example_count'}, data=[{'slice': '…

That's interesting, we only seem to have trips starting at noon on Sundays, Mondays, and Fridays!

Tracking Model Performance Over Time

Your training dataset will be used for training your model, and will hopefully be representative of your test dataset and the data that will be sent to your model in production. However, while the data in inference requests may remain the same as your training data, in many cases it will start to change enough so that the performance of your model will change.

That means that you need to monitor and measure your model's performance on an ongoing basis, so that you can be aware of and react to changes. Let's take a look at how TFMA can help.

Measure Performance For New Data

We downloaded the results of three different training runs above, so let's load them now and use TFMA to see how they compare using render_time_series. We can specify particular slices to look at. Let's compare our training runs for trips that started at noon.

  • Select a metric from the dropdown to add the time series graph for that metric
  • Close unwanted graphs
  • Hover over data points (the ends of line segments in the graph) to get more details
def get_eval_result(base_dir, output_dir, data_loc, slice_spec):
  eval_model_dir = os.path.join(base_dir, next(os.walk(base_dir))[1][0])
  eval_shared_model = tfma.default_eval_shared_model(eval_saved_model_path=eval_model_dir)

  return tfma.run_model_analysis(eval_shared_model=eval_shared_model,
                                          data_location=data_loc,
                                          file_format='tfrecords',
                                          slice_spec=slice_spec,
                                          output_path=output_dir,
                                          extractors=None)

slices = [tfma.slicer.SingleSliceSpec()]
output_dir_0 = os.path.join(TFMA_DIR, 'output', 'run_0')
result_ts0 = get_eval_result(os.path.join(TFMA_DIR, 'run_0', 'eval_model_dir'),
                             output_dir_0, TFRecord_file, slices)
output_dir_1 = os.path.join(TFMA_DIR, 'output', 'run_1')
result_ts1 = get_eval_result(os.path.join(TFMA_DIR, 'run_1', 'eval_model_dir'),
                             output_dir_1, TFRecord_file, slices)
output_dir_2 = os.path.join(TFMA_DIR, 'output', 'run_2')
result_ts2 = get_eval_result(os.path.join(TFMA_DIR, 'run_2', 'eval_model_dir'),
                             output_dir_2, TFRecord_file, slices)

How does it look today?

First, we'll imagine that we've trained and deployed our model yesterday, and now we want to see how it's doing on the new data coming in today. The visualization will start by displaying accuracy. Add AUC and average loss by using the "Add metric series" menu.

eval_results_from_disk = tfma.load_eval_results([output_dir_0, output_dir_1],
                                                tfma.constants.MODEL_CENTRIC_MODE)

tfma.view.render_time_series(eval_results_from_disk, slices[0])
TimeSeriesViewer(config={'isModelCentric': True}, data=[{'metrics': {'precision': {'doubleValue': 0.7340067625…

Now we'll imagine that another day has passed and we want to see how it's doing on the new data coming in today, compared to the previous two days. Again add AUC and average loss by using the "Add metric series" menu:

eval_results_from_disk = tfma.load_eval_results([output_dir_0, output_dir_1, output_dir_2],
                                                tfma.constants.MODEL_CENTRIC_MODE)

tfma.view.render_time_series(eval_results_from_disk, slices[0])
TimeSeriesViewer(config={'isModelCentric': True}, data=[{'metrics': {'post_export_metrics/example_count': {'do…