View on TensorFlow.org | Run in Google Colab | View on GitHub | Download notebook |
Overview
In this activity, you'll use Fairness Indicators for TensorBoard. With the plugin, you can visualize fairness evaluations for your runs and easily compare performance across groups.
Importing
Run the following code to install the required libraries.
pip install -q -U pip==20.2
pip install fairness_indicators 'absl-py<0.9,>=0.7'
pip install google-api-python-client==1.8.3
pip install tensorboard-plugin-fairness-indicators
pip install tensorflow-serving-api==2.9.0
Restart the runtime. After the runtime is restarted, continue with following cells without running previous cell again.
# %tf.disable_v2_behavior() # Uncomment this line if running in Google Colab.
import datetime
import os
import tempfile
from tensorboard_plugin_fairness_indicators import summary_v2
import tensorflow.compat.v1 as tf
# example_model.py is provided in fairness_indicators package to train and
# evaluate an example model.
from fairness_indicators import example_model
tf.compat.v1.enable_eager_execution()
2022-12-14 11:02:07.518601: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 11:02:07.518702: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 11:02:07.518712: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Data and Constants
# To know about dataset, check Fairness Indicators Example Colab at:
# https://github.com/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Fairness_Indicators_Example_Colab.ipynb
train_tf_file = tf.keras.utils.get_file('train.tf', 'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate.tf', 'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
BASE_DIR = tempfile.gettempdir()
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'
FEATURE_MAP = {
# Label:
LABEL: tf.io.FixedLenFeature([], tf.float32),
# Text:
TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),
# Identities:
'sexual_orientation': tf.io.VarLenFeature(tf.string),
'gender': tf.io.VarLenFeature(tf.string),
'religion': tf.io.VarLenFeature(tf.string),
'race': tf.io.VarLenFeature(tf.string),
'disability': tf.io.VarLenFeature(tf.string),
}
Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord 488153424/488153424 [==============================] - 12s 0us/step Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord 324941336/324941336 [==============================] - 7s 0us/step
Train the Model
model_dir = os.path.join(BASE_DIR, 'train',
datetime.datetime.now().strftime('%Y%m%d-%H%M%S'))
classifier = example_model.train_model(model_dir,
train_tf_file,
LABEL,
TEXT_FEATURE,
FEATURE_MAP)
INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmpfs/tmp/train/20221214-110229', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmpfs/tmp/train/20221214-110229', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/training/training_util.py:396: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/training/training_util.py:396: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-12-14 11:02:29.418732: W tensorflow/core/common_runtime/graph_constructor.cc:1526] Importing a graph with a lower producer version 26 into an existing graph with producer version 1286. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/canned/head.py:399: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/canned/head.py:399: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/training/adagrad.py:138: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/training/adagrad.py:138: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. 2022-12-14 11:02:32.761164: W tensorflow/core/common_runtime/type_inference.cc:339] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1: type_id: TFT_OPTIONAL args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT64 } } } is neither a subtype nor a supertype of the combined inputs preceding it: type_id: TFT_OPTIONAL args { type_id: TFT_PRODUCT args { type_id: TFT_TENSOR args { type_id: TFT_INT32 } } } while inferring type of node 'dnn/zero_fraction/cond/output/_18' INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/train/20221214-110229/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmpfs/tmp/train/20221214-110229/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 59.342094, step = 0 INFO:tensorflow:loss = 59.342094, step = 0 INFO:tensorflow:global_step/sec: 106.458 INFO:tensorflow:global_step/sec: 106.458 INFO:tensorflow:loss = 56.15521, step = 100 (0.941 sec) INFO:tensorflow:loss = 56.15521, step = 100 (0.941 sec) INFO:tensorflow:global_step/sec: 119.044 INFO:tensorflow:global_step/sec: 119.044 INFO:tensorflow:loss = 47.173832, step = 200 (0.840 sec) INFO:tensorflow:loss = 47.173832, step = 200 (0.840 sec) INFO:tensorflow:global_step/sec: 121.513 INFO:tensorflow:global_step/sec: 121.513 INFO:tensorflow:loss = 55.923626, step = 300 (0.823 sec) INFO:tensorflow:loss = 55.923626, step = 300 (0.823 sec) INFO:tensorflow:global_step/sec: 117.954 INFO:tensorflow:global_step/sec: 117.954 INFO:tensorflow:loss = 55.986916, step = 400 (0.848 sec) INFO:tensorflow:loss = 55.986916, step = 400 (0.848 sec) INFO:tensorflow:global_step/sec: 117.137 INFO:tensorflow:global_step/sec: 117.137 INFO:tensorflow:loss = 41.569267, step = 500 (0.854 sec) INFO:tensorflow:loss = 41.569267, step = 500 (0.854 sec) INFO:tensorflow:global_step/sec: 119.858 INFO:tensorflow:global_step/sec: 119.858 INFO:tensorflow:loss = 45.4774, step = 600 (0.834 sec) INFO:tensorflow:loss = 45.4774, step = 600 (0.834 sec) INFO:tensorflow:global_step/sec: 109.664 INFO:tensorflow:global_step/sec: 109.664 INFO:tensorflow:loss = 51.01483, step = 700 (0.912 sec) INFO:tensorflow:loss = 51.01483, step = 700 (0.912 sec) INFO:tensorflow:global_step/sec: 116.795 INFO:tensorflow:global_step/sec: 116.795 INFO:tensorflow:loss = 47.566914, step = 800 (0.856 sec) INFO:tensorflow:loss = 47.566914, step = 800 (0.856 sec) INFO:tensorflow:global_step/sec: 120.663 INFO:tensorflow:global_step/sec: 120.663 INFO:tensorflow:loss = 47.934036, step = 900 (0.829 sec) INFO:tensorflow:loss = 47.934036, step = 900 (0.829 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/train/20221214-110229/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmpfs/tmp/train/20221214-110229/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 51.091232. INFO:tensorflow:Loss for final step: 51.091232.
Run TensorFlow Model Analysis with Fairness Indicators
This step might take 2 to 5 minutes.
tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')
example_model.evaluate_model(classifier,
validate_tf_file,
tfma_eval_result_path,
'gender',
LABEL,
FEATURE_MAP)
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-12-14 11:02:45.003050: W tensorflow/core/common_runtime/graph_constructor.cc:1526] Importing a graph with a lower producer version 26 into an existing graph with producer version 1286. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/canned/head.py:635: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_estimator/python/estimator/canned/head.py:635: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/saved_model/model_utils/export_utils.py:84: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow/python/saved_model/model_utils/export_utils.py:84: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This API was designed for TensorFlow v1. See https://www.tensorflow.org/guide/migrate for instructions on how to migrate your code to TensorFlow v2. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmpfs/tmp/train/20221214-110229/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmpfs/tmp/train/20221214-110229/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmpfs/tmp/tfma_eval_model/temp-1671015764/assets INFO:tensorflow:Assets written to: /tmpfs/tmp/tfma_eval_model/temp-1671015764/assets INFO:tensorflow:SavedModel written to: /tmpfs/tmp/tfma_eval_model/temp-1671015764/saved_model.pb INFO:tensorflow:SavedModel written to: /tmpfs/tmp/tfma_eval_model/temp-1671015764/saved_model.pb WARNING:absl:Tensorflow version (2.11.0) found. Note that TFMA support for TF 2.0 is currently in beta WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:163: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.saved_model.load` instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:163: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.saved_model.load` instead. INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tfma_eval_model/1671015764/variables/variables INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tfma_eval_model/1671015764/variables/variables 2022-12-14 11:02:49.417145: W tensorflow/c/c_api.cc:291] Operation '{name:'dnn/metrics/auc_precision_recall/false_positives/Assign' id:1194 op device:{requested: '', assigned: ''} def:{ { {node dnn/metrics/auc_precision_recall/false_positives/Assign} } = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dnn/metrics/auc_precision_recall/false_positives, dnn/metrics/auc_precision_recall/false_positives/Initializer/zeros)} }' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session. 2022-12-14 11:02:49.766072: W tensorflow/c/c_api.cc:291] Operation '{name:'dnn/metrics/auc_precision_recall/false_positives/Assign' id:1194 op device:{requested: '', assigned: ''} def:{ { {node dnn/metrics/auc_precision_recall/false_positives/Assign} } = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](dnn/metrics/auc_precision_recall/false_positives, dnn/metrics/auc_precision_recall/false_positives/Initializer/zeros)} }' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: -*-of-%(num_shards)05d WARNING:apache_beam.io.filebasedsink:No shards found to finalize. num_shards: 1, skipped: 1 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:110: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.8/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:110: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
Visualize Fairness Indicators in TensorBoard
Below you will visualize Fairness Indicators in Tensorboard and compare performance of each slice of the data on selected metrics. You can adjust the baseline comparison slice as well as the displayed threshold(s) using the drop down menus at the top of the visualization. You can also select different evaluation runs using the drop down menu at the top-left corner.
Write Fairness Indicators Summary
Write summary file containing all required information to visualize Fairness Indicators in TensorBoard.
import tensorflow.compat.v2 as tf2
writer = tf2.summary.create_file_writer(
os.path.join(model_dir, 'fairness_indicators'))
with writer.as_default():
summary_v2.FairnessIndicators(tfma_eval_result_path, step=1)
writer.close()
Launch TensorBoard
Navigate to "Fairness Indicators" tab to visualize Fairness Indicators.
%load_ext tensorboard
%tensorboard --logdir=$model_dir