Fairness Indicators on TF-Hub Text Embeddings

View on TensorFlow.org Run in Google Colab View on GitHub Download notebook See TF Hub model

In this tutorial, you will learn how to use Fairness Indicators to evaluate embeddings from TF Hub. This notebook uses the Civil Comments dataset.

Setup

Install the required libraries.

!pip install -q fairness-indicators \
  "absl-py==0.8.0" \
  "pyarrow==0.15.1" \
  "apache-beam==2.17.0" \
  "avro-python3==1.9.1"
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

tensorflow-metadata 0.25.0 requires absl-py<0.11,>=0.9, but you'll have absl-py 0.8.0 which is incompatible.
google-api-python-client 1.12.5 requires httplib2<1dev,>=0.15.0, but you'll have httplib2 0.9.2 which is incompatible.
tfx-bsl 0.25.0 requires absl-py<0.11,>=0.9, but you'll have absl-py 0.8.0 which is incompatible.
tfx-bsl 0.25.0 requires apache-beam[gcp]<3,>=2.25, but you'll have apache-beam 2.17.0 which is incompatible.
tfx-bsl 0.25.0 requires pyarrow<0.18,>=0.17, but you'll have pyarrow 0.15.1 which is incompatible.
tensorflow-model-analysis 0.25.0 requires absl-py<0.11,>=0.9, but you'll have absl-py 0.8.0 which is incompatible.
tensorflow-model-analysis 0.25.0 requires apache-beam[gcp]<3,>=2.25, but you'll have apache-beam 2.17.0 which is incompatible.
tensorflow-model-analysis 0.25.0 requires pyarrow<0.18,>=0.17, but you'll have pyarrow 0.15.1 which is incompatible.
tensorflow-transform 0.25.0 requires absl-py<0.11,>=0.9, but you'll have absl-py 0.8.0 which is incompatible.
tensorflow-transform 0.25.0 requires apache-beam[gcp]<3,>=2.25, but you'll have apache-beam 2.17.0 which is incompatible.
tensorflow-data-validation 0.25.0 requires absl-py<0.11,>=0.9, but you'll have absl-py 0.8.0 which is incompatible.
tensorflow-data-validation 0.25.0 requires apache-beam[gcp]<3,>=2.25, but you'll have apache-beam 2.17.0 which is incompatible.
tensorflow-data-validation 0.25.0 requires pyarrow<0.18,>=0.17, but you'll have pyarrow 0.15.1 which is incompatible.
witwidget 1.7.0 requires oauth2client>=4.1.3, but you'll have oauth2client 3.0.0 which is incompatible.

Import other required libraries.

import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model

from fairness_indicators.documentation.examples import util

Dataset

In this notebook, you work with the Civil Comments dataset which contains approximately 2 million public comments made public by the Civil Comments platform in 2017 for ongoing research. This effort was sponsored by Jigsaw, who have hosted competitions on Kaggle to help classify toxic comments as well as minimize unintended model bias.

Each individual text comment in the dataset has a toxicity label, with the label being 1 if the comment is toxic and 0 if the comment is non-toxic. Within the data, a subset of comments are labeled with a variety of identity attributes, including categories for gender, sexual orientation, religion, and race or ethnicity.

Prepare the data

TensorFlow parses features from data using tf.io.FixedLenFeature and tf.io.VarLenFeature. Map out the input feature, output feature, and all other slicing features of interest.

BASE_DIR = tempfile.gettempdir()

# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'

FEATURE_MAP = {
    # input and output features
    LABEL: tf.io.FixedLenFeature([], tf.float32),
    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),

    # slicing features
    'sexual_orientation': tf.io.VarLenFeature(tf.string),
    'gender': tf.io.VarLenFeature(tf.string),
    'religion': tf.io.VarLenFeature(tf.string),
    'race': tf.io.VarLenFeature(tf.string),
    'disability': tf.io.VarLenFeature(tf.string)
}

IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']

By default, the notebook downloads a preprocessed version of this dataset, but you may use the original dataset and re-run the processing steps if desired.

In the original dataset, each comment is labeled with the percentage of raters who believed that a comment corresponds to a particular identity. For example, a comment might be labeled with the following: { male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0 }.

The processing step groups identity by category (gender, sexual_orientation, etc.) and removes identities with a score less than 0.5. So the example above would be converted to the following: of raters who believed that a comment corresponds to a particular identity. For example, the comment above would be labeled with the following: { gender: [female], sexual_orientation: [heterosexual, homosexual_gay_or_lesbian] }

Download the dataset.

download_original_data = False

if download_original_data:
  train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')

  # The identity terms list will be grouped together by their categories
  # (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,
  # text column and label column will be kept after processing.
  train_tf_file = util.convert_comments_data(train_tf_file)
  validate_tf_file = util.convert_comments_data(validate_tf_file)

else:
  train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')

Create a TensorFlow Model Analysis Pipeline

The Fairness Indicators library operates on TensorFlow Model Analysis (TFMA) models. TFMA models wrap TensorFlow models with additional functionality to evaluate and visualize their results. The actual evaluation occurs inside of an Apache Beam pipeline.

The steps you follow to create a TFMA pipeline are:

  1. Build a TensorFlow model
  2. Build a TFMA model on top of the TensorFlow model
  3. Run the model analysis in an orchestrator. The example model in this notebook uses Apache Beam as the orchestrator.
def embedding_fairness_result(embedding, identity_term='gender'):

  model_dir = os.path.join(BASE_DIR, 'train',
                         datetime.now().strftime('%Y%m%d-%H%M%S'))

  print("Training classifier for " + embedding)
  classifier = example_model.train_model(model_dir,
                                         train_tf_file,
                                         LABEL,
                                         TEXT_FEATURE,
                                         FEATURE_MAP,
                                         embedding)

  # Create a unique path to store the results for this embedding.
  embedding_name = embedding.split('/')[-2]
  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)

  example_model.evaluate_model(classifier,
                               validate_tf_file,
                               eval_result_path,
                               identity_term,
                               LABEL,
                               FEATURE_MAP)
  return tfma.load_eval_result(output_path=eval_result_path)

Run TFMA & Fairness Indicators

Fairness Indicators Metrics

Some of the metrics available with Fairness Indicators are:

Text Embeddings

TF-Hub provides several text embeddings. These embeddings will serve as the feature column for the different models. This tutorial uses the following embeddings:

Fairness Indicator Results

Compute fairness indicators with the embedding_fairness_result pipeline, and then render the results in the Fairness Indicator UI widget with widget_view.render_fairness_indicator for all the above embeddings.

Random NNLM

eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/random-nnlm-en-dim128/1
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20201116-014902', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20201116-014902', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20201116-014902/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20201116-014902/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 63.290295, step = 0

INFO:tensorflow:loss = 63.290295, step = 0

INFO:tensorflow:global_step/sec: 22.0479

INFO:tensorflow:global_step/sec: 22.0479

INFO:tensorflow:loss = 68.421104, step = 100 (4.537 sec)

INFO:tensorflow:loss = 68.421104, step = 100 (4.537 sec)

INFO:tensorflow:global_step/sec: 21.615

INFO:tensorflow:global_step/sec: 21.615

INFO:tensorflow:loss = 59.118176, step = 200 (4.628 sec)

INFO:tensorflow:loss = 59.118176, step = 200 (4.628 sec)

INFO:tensorflow:global_step/sec: 22.9624

INFO:tensorflow:global_step/sec: 22.9624

INFO:tensorflow:loss = 59.982048, step = 300 (4.353 sec)

INFO:tensorflow:loss = 59.982048, step = 300 (4.353 sec)

INFO:tensorflow:global_step/sec: 22.5573

INFO:tensorflow:global_step/sec: 22.5573

INFO:tensorflow:loss = 61.75359, step = 400 (4.433 sec)

INFO:tensorflow:loss = 61.75359, step = 400 (4.433 sec)

INFO:tensorflow:global_step/sec: 22.4348

INFO:tensorflow:global_step/sec: 22.4348

INFO:tensorflow:loss = 55.901016, step = 500 (4.457 sec)

INFO:tensorflow:loss = 55.901016, step = 500 (4.457 sec)

INFO:tensorflow:global_step/sec: 22.0914

INFO:tensorflow:global_step/sec: 22.0914

INFO:tensorflow:loss = 57.899063, step = 600 (4.527 sec)

INFO:tensorflow:loss = 57.899063, step = 600 (4.527 sec)

INFO:tensorflow:global_step/sec: 22.4452

INFO:tensorflow:global_step/sec: 22.4452

INFO:tensorflow:loss = 61.906555, step = 700 (4.455 sec)

INFO:tensorflow:loss = 61.906555, step = 700 (4.455 sec)

INFO:tensorflow:global_step/sec: 22.2387

INFO:tensorflow:global_step/sec: 22.2387

INFO:tensorflow:loss = 57.230656, step = 800 (4.497 sec)

INFO:tensorflow:loss = 57.230656, step = 800 (4.497 sec)

INFO:tensorflow:global_step/sec: 22.6482

INFO:tensorflow:global_step/sec: 22.6482

INFO:tensorflow:loss = 56.52168, step = 900 (4.418 sec)

INFO:tensorflow:loss = 56.52168, step = 900 (4.418 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20201116-014902/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20201116-014902/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Loss for final step: 59.80134.

INFO:tensorflow:Loss for final step: 59.80134.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

Warning:tensorflow:Export includes no default signature!

Warning:tensorflow:Export includes no default signature!

INFO:tensorflow:Restoring parameters from /tmp/train/20201116-014902/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmp/train/20201116-014902/model.ckpt-1000

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1605491402/assets

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1605491402/assets

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1605491402/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1605491402/saved_model.pb
WARNING:absl:Tensorflow version (2.3.1) found. Note that TFMA support for TF 2.0 is currently in beta

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1605491402/variables/variables

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1605491402/variables/variables

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:root:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`

Warning:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`

widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

NNLM

eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/nnlm-en-dim128/1
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20201116-015221', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20201116-015221', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20201116-015221/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20201116-015221/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 59.183952, step = 0

INFO:tensorflow:loss = 59.183952, step = 0

INFO:tensorflow:global_step/sec: 18.715

INFO:tensorflow:global_step/sec: 18.715

INFO:tensorflow:loss = 56.49299, step = 100 (5.346 sec)

INFO:tensorflow:loss = 56.49299, step = 100 (5.346 sec)

INFO:tensorflow:global_step/sec: 15.5656

INFO:tensorflow:global_step/sec: 15.5656

INFO:tensorflow:loss = 47.24923, step = 200 (6.425 sec)

INFO:tensorflow:loss = 47.24923, step = 200 (6.425 sec)

INFO:tensorflow:global_step/sec: 20.3296

INFO:tensorflow:global_step/sec: 20.3296

INFO:tensorflow:loss = 55.820904, step = 300 (4.918 sec)

INFO:tensorflow:loss = 55.820904, step = 300 (4.918 sec)

INFO:tensorflow:global_step/sec: 21.9064

INFO:tensorflow:global_step/sec: 21.9064

INFO:tensorflow:loss = 55.29339, step = 400 (4.565 sec)

INFO:tensorflow:loss = 55.29339, step = 400 (4.565 sec)

INFO:tensorflow:global_step/sec: 22.0037

INFO:tensorflow:global_step/sec: 22.0037

INFO:tensorflow:loss = 41.25977, step = 500 (4.545 sec)

INFO:tensorflow:loss = 41.25977, step = 500 (4.545 sec)

INFO:tensorflow:global_step/sec: 22.1313

INFO:tensorflow:global_step/sec: 22.1313

INFO:tensorflow:loss = 45.79728, step = 600 (4.519 sec)

INFO:tensorflow:loss = 45.79728, step = 600 (4.519 sec)

INFO:tensorflow:global_step/sec: 22.0979

INFO:tensorflow:global_step/sec: 22.0979

INFO:tensorflow:loss = 50.843258, step = 700 (4.524 sec)

INFO:tensorflow:loss = 50.843258, step = 700 (4.524 sec)

INFO:tensorflow:global_step/sec: 22.1194

INFO:tensorflow:global_step/sec: 22.1194

INFO:tensorflow:loss = 47.699226, step = 800 (4.521 sec)

INFO:tensorflow:loss = 47.699226, step = 800 (4.521 sec)

INFO:tensorflow:global_step/sec: 22.097

INFO:tensorflow:global_step/sec: 22.097

INFO:tensorflow:loss = 48.315933, step = 900 (4.525 sec)

INFO:tensorflow:loss = 48.315933, step = 900 (4.525 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20201116-015221/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20201116-015221/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Loss for final step: 50.75473.

INFO:tensorflow:Loss for final step: 50.75473.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

Warning:tensorflow:Export includes no default signature!

Warning:tensorflow:Export includes no default signature!

INFO:tensorflow:Restoring parameters from /tmp/train/20201116-015221/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmp/train/20201116-015221/model.ckpt-1000

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1605491593/assets

INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1605491593/assets

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1605491593/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1605491593/saved_model.pb
WARNING:absl:Tensorflow version (2.3.1) found. Note that TFMA support for TF 2.0 is currently in beta

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1605491593/variables/variables

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1605491593/variables/variables

widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

Universal Sentence Encoder

eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')
Training classifier for https://tfhub.dev/google/universal-sentence-encoder/2
INFO:tensorflow:Using default config.

INFO:tensorflow:Using default config.

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20201116-015533', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20201116-015533', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Graph was finalized.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20201116-015533/model.ckpt.

INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20201116-015533/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...

INFO:tensorflow:loss = 58.963093, step = 0

INFO:tensorflow:loss = 58.963093, step = 0

INFO:tensorflow:global_step/sec: 8.04923

INFO:tensorflow:global_step/sec: 8.04923

INFO:tensorflow:loss = 49.000237, step = 100 (12.425 sec)

INFO:tensorflow:loss = 49.000237, step = 100 (12.425 sec)

INFO:tensorflow:global_step/sec: 8.30145

INFO:tensorflow:global_step/sec: 8.30145

INFO:tensorflow:loss = 45.99742, step = 200 (12.046 sec)

INFO:tensorflow:loss = 45.99742, step = 200 (12.046 sec)

INFO:tensorflow:global_step/sec: 8.24003

INFO:tensorflow:global_step/sec: 8.24003

INFO:tensorflow:loss = 47.92001, step = 300 (12.136 sec)

INFO:tensorflow:loss = 47.92001, step = 300 (12.136 sec)

INFO:tensorflow:global_step/sec: 8.2365

INFO:tensorflow:global_step/sec: 8.2365

INFO:tensorflow:loss = 44.52539, step = 400 (12.141 sec)

INFO:tensorflow:loss = 44.52539, step = 400 (12.141 sec)

INFO:tensorflow:global_step/sec: 8.28065

INFO:tensorflow:global_step/sec: 8.28065

INFO:tensorflow:loss = 35.426815, step = 500 (12.077 sec)

INFO:tensorflow:loss = 35.426815, step = 500 (12.077 sec)

INFO:tensorflow:global_step/sec: 8.35115

INFO:tensorflow:global_step/sec: 8.35115

INFO:tensorflow:loss = 42.07934, step = 600 (11.974 sec)

INFO:tensorflow:loss = 42.07934, step = 600 (11.974 sec)

INFO:tensorflow:global_step/sec: 8.24161

INFO:tensorflow:global_step/sec: 8.24161

INFO:tensorflow:loss = 40.6244, step = 700 (12.134 sec)

INFO:tensorflow:loss = 40.6244, step = 700 (12.134 sec)

INFO:tensorflow:global_step/sec: 8.02892

INFO:tensorflow:global_step/sec: 8.02892

INFO:tensorflow:loss = 37.263874, step = 800 (12.455 sec)

INFO:tensorflow:loss = 37.263874, step = 800 (12.455 sec)

INFO:tensorflow:global_step/sec: 8.35795

INFO:tensorflow:global_step/sec: 8.35795

INFO:tensorflow:loss = 32.879314, step = 900 (11.965 sec)

INFO:tensorflow:loss = 32.879314, step = 900 (11.965 sec)

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20201116-015533/model.ckpt.

INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20201116-015533/model.ckpt.

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...

INFO:tensorflow:Loss for final step: 47.237846.

INFO:tensorflow:Loss for final step: 47.237846.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Calling model_fn.

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

INFO:tensorflow:Saver not created because there are no variables in the graph to restore

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

Warning:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Classify: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Regress: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Predict: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Train: None

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']

Warning:tensorflow:Export includes no default signature!

Warning:tensorflow:Export includes no default signature!

INFO:tensorflow:Restoring parameters from /tmp/train/20201116-015533/model.ckpt-1000

INFO:tensorflow:Restoring parameters from /tmp/train/20201116-015533/model.ckpt-1000

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:Assets added to graph.

INFO:tensorflow:No assets to write.

INFO:tensorflow:No assets to write.

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1605491895/saved_model.pb

INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1605491895/saved_model.pb
WARNING:absl:Tensorflow version (2.3.1) found. Note that TFMA support for TF 2.0 is currently in beta

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1605491895/variables/variables

INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1605491895/variables/variables

widget_view.render_fairness_indicator(eval_result=eval_result_use)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

Comparing Embeddings

You can also use Fairness Indicators to compare embeddings directly. For example, compare the models generated from the NNLM and USE embeddings.

widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})
FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…