Fairness Indicators on TF-Hub Text Embeddings

In this colab, you will learn how to use Fairness Indicators to evaluate embeddings from TF Hub. Fairness Indicators is a suite of tools that facilitates evaluation and visualization of fairness metrics on machine learning models. Fairness Indicators is built on top of TensorFlow Model Analysis, TensorFlow's official model evaluation library.

Imports

!pip install -q fairness-indicators \
  "absl-py==0.8.0" \
  "pyarrow==0.15.1" \
  "apache-beam==2.17.0" \
  "avro-python3==1.9.1"
ERROR: witwidget 1.6.0 has requirement oauth2client>=4.1.3, but you'll have oauth2client 3.0.0 which is incompatible.
ERROR: tensorflow-serving-api 2.1.0 has requirement tensorflow~=2.1.0, but you'll have tensorflow 2.2.0 which is incompatible.
ERROR: tfx-bsl 0.22.0 has requirement apache-beam[gcp]<3,>=2.20, but you'll have apache-beam 2.17.0 which is incompatible.
ERROR: tfx-bsl 0.22.0 has requirement pyarrow<0.17,>=0.16.0, but you'll have pyarrow 0.15.1 which is incompatible.
ERROR: tensorflow-model-analysis 0.22.0 has requirement apache-beam[gcp]<3,>=2.20, but you'll have apache-beam 2.17.0 which is incompatible.
ERROR: tensorflow-model-analysis 0.22.0 has requirement pyarrow<1,>=0.16, but you'll have pyarrow 0.15.1 which is incompatible.
ERROR: tensorflow-transform 0.21.2 has requirement tensorflow<2.2,>=1.15, but you'll have tensorflow 2.2.0 which is incompatible.
ERROR: tensorflow-transform 0.21.2 has requirement tensorflow-metadata<0.22,>=0.21, but you'll have tensorflow-metadata 0.22.0 which is incompatible.
ERROR: tensorflow-transform 0.21.2 has requirement tfx-bsl<0.22,>=0.21.3, but you'll have tfx-bsl 0.22.0 which is incompatible.
ERROR: tensorflow-data-validation 0.21.5 has requirement pandas<1,>=0.24, but you'll have pandas 1.0.3 which is incompatible.
ERROR: tensorflow-data-validation 0.21.5 has requirement tensorflow-metadata<0.22,>=0.21.1, but you'll have tensorflow-metadata 0.22.0 which is incompatible.
ERROR: tensorflow-data-validation 0.21.5 has requirement tfx-bsl<0.22,>=0.21.3, but you'll have tfx-bsl 0.22.0 which is incompatible.

import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model
from fairness_indicators.examples import util
Error importing tfx_bsl_extension.coders. Some tfx_bsl functionalities are not availableError importing tfx_bsl_extension.arrow.array_util. Some tfx_bsl functionalities are not availableError importing tfx_bsl_extension.arrow.table_util. Some tfx_bsl functionalities are not available: libarrow.so.16: cannot open shared object file: No such file or directory

Defining Constants

TensorFlow parses features from data using FixedLenFeature and VarLenFeature. So to allow TensorFlow to parse our data, we will need to map out our input feature, output feature, and any slicing features that we will want to analyze via Fairness Indicators.

BASE_DIR = tempfile.gettempdir()

# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'

FEATURE_MAP = {
    # input and output features
    LABEL: tf.io.FixedLenFeature([], tf.float32),
    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),

    # slicing features
    'sexual_orientation': tf.io.VarLenFeature(tf.string),
    'gender': tf.io.VarLenFeature(tf.string),
    'religion': tf.io.VarLenFeature(tf.string),
    'race': tf.io.VarLenFeature(tf.string),
    'disability': tf.io.VarLenFeature(tf.string)
}

IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']

Data

In this exercise, we'll work with the Civil Comments dataset, approximately 2 million public comments made public by the Civil Comments platform in 2017 for ongoing research. This effort was sponsored by Jigsaw, who have hosted competitions on Kaggle to help classify toxic comments as well as minimize unintended model bias.

Each individual text comment in the dataset has a toxicity label, with the label being 1 if the comment is toxic and 0 if the comment is non-toxic. Within the data, a subset of comments are labeled with a variety of identity attributes, including categories for gender, sexual orientation, religion, and race or ethnicity.

You can choose to download the original dataset and process it in the colab, which may take minutes, or you can download the preprocessed data.

download_original_data = True

if download_original_data:
  train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')

  # The identity terms list will be grouped together by their categories
  # on threshould 0.5. Only the identity term column, text column,
  # and label column will be kept after processing.
  train_tf_file = util.convert_comments_data(train_tf_file)
  validate_tf_file = util.convert_comments_data(validate_tf_file)

else:
  train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord
1439031296/1439024821 [==============================] - 23s 0us/step
Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord
958767104/958765415 [==============================] - 14s 0us/step

Creating a TensorFlow Model Analysis Pipeline

The Fairness Indicators library operates on TensorFlow Model Analysis (TFMA) models. TFMA models wrap TensorFlow models with additional functionality to evaluate and visualize their results. The actual evaluation occurs inside of an Apache Beam pipeline.

So we need to...

  1. Build a TensorFlow model.
  2. Build a TFMA model on top of the TensorFlow model.
  3. Run the model analysis in a Beam pipeline.

Putting it all Together

def embedding_fairness_result(embedding, identity_term='gender'):
  
  model_dir = os.path.join(BASE_DIR, 'train',
                         datetime.now().strftime('%Y%m%d-%H%M%S'))

  print("Training classifier for " + embedding)
  classifier = example_model.train_model(model_dir,
                                         train_tf_file,
                                         LABEL,
                                         TEXT_FEATURE,
                                         FEATURE_MAP,
                                         embedding)

  # We need to create a unique path to store our results for this embedding.
  embedding_name = embedding.split('/')[-2]
  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)

  example_model.evaluate_model(classifier,
                               validate_tf_file,
                               eval_result_path,
                               identity_term,
                               LABEL,
                               FEATURE_MAP)
  return tfma.load_eval_result(output_path=eval_result_path)

Run TFMA & Fairness Indicators

Fairness Indicators Metrics

Refer here for more information on Fairness Indicators. Below are some of the available metrics.

Text Embeddings

TF-Hub provides several text embeddings. These embeddings will serve as the feature column for our different models. For this Colab, we use the following embeddings:

Fairness Indicator Results

For each of the above embeddings, we will compute fairness indicators with our embedding_fairness_result pipeline, and then render the results in the Fairness Indicator UI widget with widget_view.render_fairness_indicator.

Note that the widget_view.render_fairness_indicator cells may need to be run twice for the visualization to be displayed.

Random NNLM

eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/random-nnlm-en-dim128/1
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20200514-094028', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20200514-094028', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:1666: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2167: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2167: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20200514-094028/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20200514-094028/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 65.96399, step = 0

INFO:tensorflow:loss = 65.96399, step = 0

INFO:tensorflow:global_step/sec: 22.9274

INFO:tensorflow:global_step/sec: 22.9274

INFO:tensorflow:loss = 69.29546, step = 100 (4.364 sec)

INFO:tensorflow:loss = 69.29546, step = 100 (4.364 sec)

INFO:tensorflow:global_step/sec: 23.265

INFO:tensorflow:global_step/sec: 23.265

INFO:tensorflow:loss = 59.74574, step = 200 (4.298 sec)

INFO:tensorflow:loss = 59.74574, step = 200 (4.298 sec)

INFO:tensorflow:global_step/sec: 22.851

INFO:tensorflow:global_step/sec: 22.851

INFO:tensorflow:loss = 60.279285, step = 300 (4.376 sec)

INFO:tensorflow:loss = 60.279285, step = 300 (4.376 sec)

INFO:tensorflow:global_step/sec: 23.0458

INFO:tensorflow:global_step/sec: 23.0458

INFO:tensorflow:loss = 61.069134, step = 400 (4.339 sec)

INFO:tensorflow:loss = 61.069134, step = 400 (4.339 sec)

INFO:tensorflow:global_step/sec: 23.2022

INFO:tensorflow:global_step/sec: 23.2022

INFO:tensorflow:loss = 56.2027, step = 500 (4.310 sec)

INFO:tensorflow:loss = 56.2027, step = 500 (4.310 sec)

INFO:tensorflow:global_step/sec: 22.2797

INFO:tensorflow:global_step/sec: 22.2797

INFO:tensorflow:loss = 57.626698, step = 600 (4.488 sec)

INFO:tensorflow:loss = 57.626698, step = 600 (4.488 sec)

INFO:tensorflow:global_step/sec: 23.2264

INFO:tensorflow:global_step/sec: 23.2264

INFO:tensorflow:loss = 61.40799, step = 700 (4.305 sec)

INFO:tensorflow:loss = 61.40799, step = 700 (4.305 sec)

INFO:tensorflow:global_step/sec: 22.6711

INFO:tensorflow:global_step/sec: 22.6711

INFO:tensorflow:loss = 57.562744, step = 800 (4.411 sec)

INFO:tensorflow:loss = 57.562744, step = 800 (4.411 sec)

INFO:tensorflow:global_step/sec: 23.1929

INFO:tensorflow:global_step/sec: 23.1929

INFO:tensorflow:loss = 57.97668, step = 900 (4.311 sec)

INFO:tensorflow:loss = 57.97668, step = 900 (4.311 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20200514-094028/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20200514-094028/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Loss for final step: 60.242737.

INFO:tensorflow:Loss for final step: 60.242737.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

Warning:tensorflow:Export includes no default signature!

Warning:tensorflow:Export includes no default signature!

INFO:tensorflow:Restoring parameters from /tmp/train/20200514-094028/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmp/train/20200514-094028/model.ckpt-1000

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1589449284/assets

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1589449284/assets

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1589449284/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1589449284/saved_model.pb
WARNING:absl:Tensorflow version (2.2.0) found. Note that TFMA support for TF 2.0 is currently in beta

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1589449284/variables/variables

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1589449284/variables/variables

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:root:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_and_plots_serialization.py:122: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_and_plots_serialization.py:122: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`

widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

NNLM

eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/nnlm-en-dim128/1
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20200514-094317', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20200514-094317', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20200514-094317/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20200514-094317/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 60.258698, step = 0

INFO:tensorflow:loss = 60.258698, step = 0

INFO:tensorflow:global_step/sec: 21.979

INFO:tensorflow:global_step/sec: 21.979

INFO:tensorflow:loss = 56.129803, step = 100 (4.552 sec)

INFO:tensorflow:loss = 56.129803, step = 100 (4.552 sec)

INFO:tensorflow:global_step/sec: 22.4693

INFO:tensorflow:global_step/sec: 22.4693

INFO:tensorflow:loss = 47.49465, step = 200 (4.451 sec)

INFO:tensorflow:loss = 47.49465, step = 200 (4.451 sec)

INFO:tensorflow:global_step/sec: 23.0959

INFO:tensorflow:global_step/sec: 23.0959

INFO:tensorflow:loss = 56.44122, step = 300 (4.330 sec)

INFO:tensorflow:loss = 56.44122, step = 300 (4.330 sec)

INFO:tensorflow:global_step/sec: 21.2511

INFO:tensorflow:global_step/sec: 21.2511

INFO:tensorflow:loss = 55.66918, step = 400 (4.706 sec)

INFO:tensorflow:loss = 55.66918, step = 400 (4.706 sec)

INFO:tensorflow:global_step/sec: 16.3321

INFO:tensorflow:global_step/sec: 16.3321

INFO:tensorflow:loss = 41.659523, step = 500 (6.123 sec)

INFO:tensorflow:loss = 41.659523, step = 500 (6.123 sec)

INFO:tensorflow:global_step/sec: 18.7305

INFO:tensorflow:global_step/sec: 18.7305

INFO:tensorflow:loss = 45.635, step = 600 (5.339 sec)

INFO:tensorflow:loss = 45.635, step = 600 (5.339 sec)

INFO:tensorflow:global_step/sec: 22.2569

INFO:tensorflow:global_step/sec: 22.2569

INFO:tensorflow:loss = 50.95803, step = 700 (4.493 sec)

INFO:tensorflow:loss = 50.95803, step = 700 (4.493 sec)

INFO:tensorflow:global_step/sec: 22.9371

INFO:tensorflow:global_step/sec: 22.9371

INFO:tensorflow:loss = 47.713024, step = 800 (4.360 sec)

INFO:tensorflow:loss = 47.713024, step = 800 (4.360 sec)

INFO:tensorflow:global_step/sec: 23.4985

INFO:tensorflow:global_step/sec: 23.4985

INFO:tensorflow:loss = 48.284737, step = 900 (4.256 sec)

INFO:tensorflow:loss = 48.284737, step = 900 (4.256 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20200514-094317/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20200514-094317/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Loss for final step: 50.978153.

INFO:tensorflow:Loss for final step: 50.978153.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

Warning:tensorflow:Export includes no default signature!

Warning:tensorflow:Export includes no default signature!

INFO:tensorflow:Restoring parameters from /tmp/train/20200514-094317/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmp/train/20200514-094317/model.ckpt-1000

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1589449455/assets

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1589449455/assets

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1589449455/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1589449455/saved_model.pb
WARNING:absl:Tensorflow version (2.2.0) found. Note that TFMA support for TF 2.0 is currently in beta

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1589449455/variables/variables

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1589449455/variables/variables

widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

Universal Sentence Encoder

eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')
Training classifier for https://tfhub.dev/google/universal-sentence-encoder/2
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20200514-094608', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20200514-094608', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20200514-094608/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20200514-094608/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 59.179375, step = 0

INFO:tensorflow:loss = 59.179375, step = 0

INFO:tensorflow:global_step/sec: 8.80147

INFO:tensorflow:global_step/sec: 8.80147

INFO:tensorflow:loss = 49.14518, step = 100 (11.363 sec)

INFO:tensorflow:loss = 49.14518, step = 100 (11.363 sec)

INFO:tensorflow:global_step/sec: 9.17873

INFO:tensorflow:global_step/sec: 9.17873

INFO:tensorflow:loss = 46.454334, step = 200 (10.895 sec)

INFO:tensorflow:loss = 46.454334, step = 200 (10.895 sec)

INFO:tensorflow:global_step/sec: 8.92924

INFO:tensorflow:global_step/sec: 8.92924

INFO:tensorflow:loss = 48.14746, step = 300 (11.199 sec)

INFO:tensorflow:loss = 48.14746, step = 300 (11.199 sec)

INFO:tensorflow:global_step/sec: 9.19174

INFO:tensorflow:global_step/sec: 9.19174

INFO:tensorflow:loss = 44.894608, step = 400 (10.880 sec)

INFO:tensorflow:loss = 44.894608, step = 400 (10.880 sec)

INFO:tensorflow:global_step/sec: 9.26001

INFO:tensorflow:global_step/sec: 9.26001

INFO:tensorflow:loss = 35.484497, step = 500 (10.799 sec)

INFO:tensorflow:loss = 35.484497, step = 500 (10.799 sec)

INFO:tensorflow:global_step/sec: 9.20132

INFO:tensorflow:global_step/sec: 9.20132

INFO:tensorflow:loss = 42.30106, step = 600 (10.868 sec)

INFO:tensorflow:loss = 42.30106, step = 600 (10.868 sec)

INFO:tensorflow:global_step/sec: 8.9834

INFO:tensorflow:global_step/sec: 8.9834

INFO:tensorflow:loss = 40.877647, step = 700 (11.132 sec)

INFO:tensorflow:loss = 40.877647, step = 700 (11.132 sec)

INFO:tensorflow:global_step/sec: 9.20794

INFO:tensorflow:global_step/sec: 9.20794

INFO:tensorflow:loss = 37.46688, step = 800 (10.860 sec)

INFO:tensorflow:loss = 37.46688, step = 800 (10.860 sec)

INFO:tensorflow:global_step/sec: 9.24414

INFO:tensorflow:global_step/sec: 9.24414

INFO:tensorflow:loss = 33.13515, step = 900 (10.818 sec)

INFO:tensorflow:loss = 33.13515, step = 900 (10.818 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20200514-094608/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20200514-094608/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Loss for final step: 47.03059.

INFO:tensorflow:Loss for final step: 47.03059.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

Warning:tensorflow:Export includes no default signature!

Warning:tensorflow:Export includes no default signature!

INFO:tensorflow:Restoring parameters from /tmp/train/20200514-094608/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmp/train/20200514-094608/model.ckpt-1000

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1589449722/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1589449722/saved_model.pb
WARNING:absl:Tensorflow version (2.2.0) found. Note that TFMA support for TF 2.0 is currently in beta

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1589449722/variables/variables

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1589449722/variables/variables

widget_view.render_fairness_indicator(eval_result=eval_result_use)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

Comparing Embeddings

We can also use Fairness Indicators to compare embeddings directly. Let's compare the models generated from the NNLM and USE embeddings.

widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})
FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…

Exercises

  1. Pick an identity category, such as religion or sexual orientation, and look at False Positive Rate for the Universal Sentence Encoder. How do different slices compare to each other? How do they compare to the Overall baseline?
  2. Now pick a different identity category. Compare the results of this category with the previous one. Does the model weigh one category as more "toxic" than the other? Does this change with the embedding used?
  3. Does the model generally tend to overestimate or underestimate the number of toxic comments?
  4. Look at the graphs for different fairness metrics. Which metrics seem most informative? Which embeddings perform best and worst for that metric?