مشاهده در TensorFlow.org

در Google Colab اجرا شود

در GitHub مشاهده کنید

دانلود دفترچه یادداشت

بررسی اجمالی

در این فعالیت، شما با استفاده از شاخص عدالت برای TensorBoard . با این افزونه، می‌توانید ارزیابی‌های عادلانه را برای اجراهای خود تجسم کنید و به راحتی عملکرد را در گروه‌ها مقایسه کنید.

واردات

کد زیر را برای نصب کتابخانه های مورد نیاز اجرا کنید.

pip install -q -U pip==20.2

pip install fairness_indicators 'absl-py<0.9,>=0.7'
pip install google-api-python-client==1.8.3
pip install tensorboard-plugin-fairness-indicators
pip install tensorflow-serving-api==2.7.0

زمان اجرا را مجدداً راه اندازی کنید. پس از شروع مجدد زمان اجرا، بدون اجرای مجدد سلول قبلی، به سلول های زیر ادامه دهید.

# %tf.disable_v2_behavior() # Uncomment this line if running in Google Colab.

import datetime
import os
import tempfile
from tensorboard_plugin_fairness_indicators import summary_v2
import tensorflow.compat.v1 as tf

# example_model.py is provided in fairness_indicators package to train and
# evaluate an example model. 
from fairness_indicators import example_model

tf.compat.v1.enable_eager_execution()

داده ها و ثابت ها

# To know about dataset, check Fairness Indicators Example Colab at:
# https://github.com/tensorflow/fairness-indicators/blob/master/g3doc/tutorials/Fairness_Indicators_Example_Colab.ipynb

train_tf_file = tf.keras.utils.get_file('train.tf', 'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate.tf', 'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')

BASE_DIR = tempfile.gettempdir()
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'
FEATURE_MAP = {
    # Label:
    LABEL: tf.io.FixedLenFeature([], tf.float32),
    # Text:
    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),

    # Identities:
    'sexual_orientation': tf.io.VarLenFeature(tf.string),
    'gender': tf.io.VarLenFeature(tf.string),
    'religion': tf.io.VarLenFeature(tf.string),
    'race': tf.io.VarLenFeature(tf.string),
    'disability': tf.io.VarLenFeature(tf.string),
}

Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord
488161280/488153424 [==============================] - 11s 0us/step
488169472/488153424 [==============================] - 11s 0us/step
Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord
324943872/324941336 [==============================] - 9s 0us/step
324952064/324941336 [==============================] - 9s 0us/step

مدل را آموزش دهید

model_dir = os.path.join(BASE_DIR, 'train',
                         datetime.datetime.now().strftime('%Y%m%d-%H%M%S'))

classifier = example_model.train_model(model_dir,
                                       train_tf_file,
                                       LABEL,
                                       TEXT_FEATURE,
                                       FEATURE_MAP)

INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-180912', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-180912', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:09:22.433489: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-180912/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-180912/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 58.65023, step = 0
INFO:tensorflow:loss = 58.65023, step = 0
INFO:tensorflow:global_step/sec: 76.6785
INFO:tensorflow:global_step/sec: 76.6785
INFO:tensorflow:loss = 55.854782, step = 100 (1.306 sec)
INFO:tensorflow:loss = 55.854782, step = 100 (1.306 sec)
INFO:tensorflow:global_step/sec: 82.6
INFO:tensorflow:global_step/sec: 82.6
INFO:tensorflow:loss = 47.47064, step = 200 (1.210 sec)
INFO:tensorflow:loss = 47.47064, step = 200 (1.210 sec)
INFO:tensorflow:global_step/sec: 85.8208
INFO:tensorflow:global_step/sec: 85.8208
INFO:tensorflow:loss = 55.59231, step = 300 (1.166 sec)
INFO:tensorflow:loss = 55.59231, step = 300 (1.166 sec)
INFO:tensorflow:global_step/sec: 86.1252
INFO:tensorflow:global_step/sec: 86.1252
INFO:tensorflow:loss = 56.18415, step = 400 (1.161 sec)
INFO:tensorflow:loss = 56.18415, step = 400 (1.161 sec)
INFO:tensorflow:global_step/sec: 86.5735
INFO:tensorflow:global_step/sec: 86.5735
INFO:tensorflow:loss = 42.37696, step = 500 (1.155 sec)
INFO:tensorflow:loss = 42.37696, step = 500 (1.155 sec)
INFO:tensorflow:global_step/sec: 86.6948
INFO:tensorflow:global_step/sec: 86.6948
INFO:tensorflow:loss = 45.75257, step = 600 (1.153 sec)
INFO:tensorflow:loss = 45.75257, step = 600 (1.153 sec)
INFO:tensorflow:global_step/sec: 87.1878
INFO:tensorflow:global_step/sec: 87.1878
INFO:tensorflow:loss = 50.73873, step = 700 (1.147 sec)
INFO:tensorflow:loss = 50.73873, step = 700 (1.147 sec)
INFO:tensorflow:global_step/sec: 85.2284
INFO:tensorflow:global_step/sec: 85.2284
INFO:tensorflow:loss = 47.609695, step = 800 (1.173 sec)
INFO:tensorflow:loss = 47.609695, step = 800 (1.173 sec)
INFO:tensorflow:global_step/sec: 85.6373
INFO:tensorflow:global_step/sec: 85.6373
INFO:tensorflow:loss = 48.22233, step = 900 (1.168 sec)
INFO:tensorflow:loss = 48.22233, step = 900 (1.168 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-180912/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-180912/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 51.06088.
INFO:tensorflow:Loss for final step: 51.06088.

تجزیه و تحلیل مدل TensorFlow را با شاخص های انصاف اجرا کنید

این مرحله ممکن است 2 تا 5 دقیقه طول بکشد.

tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')

example_model.evaluate_model(classifier,
                             validate_tf_file,
                             tfma_eval_result_path,
                             'gender',
                             LABEL,
                             FEATURE_MAP)

WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:09:39.350323: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-180912/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-180912/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641578979/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641578979/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641578979/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641578979/saved_model.pb
WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641578979/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641578979/variables/variables
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`

نشانگرهای انصاف را در TensorBoard تجسم کنید

در زیر شما نشانگرهای انصاف را در Tensorboard تجسم می‌کنید و عملکرد هر بخش از داده‌ها را در معیارهای انتخابی مقایسه می‌کنید. می توانید با استفاده از منوهای کشویی در بالای تجسم، برش مقایسه پایه و همچنین آستانه(های) نمایش داده شده را تنظیم کنید. همچنین می توانید با استفاده از منوی کشویی در گوشه بالا سمت چپ، مراحل ارزیابی مختلف را انتخاب کنید.

خلاصه شاخص های انصاف را بنویسید

فایل خلاصه ای بنویسید که حاوی تمام اطلاعات لازم برای تجسم نشانگرهای عدالت در TensorBoard باشد.

import tensorflow.compat.v2 as tf2

writer = tf2.summary.create_file_writer(
    os.path.join(model_dir, 'fairness_indicators'))
with writer.as_default():
  summary_v2.FairnessIndicators(tfma_eval_result_path, step=1)
writer.close()

TensorBoard را راه اندازی کنید

برای تجسم نشانگرهای انصاف، به تب "شاخص های انصاف" بروید.

%load_ext tensorboard

%tensorboard --logdir=$model_dir