مؤشرات الإنصاف على TF-Hub Text Embeddings

عرض على TensorFlow.org

تشغيل في Google Colab

عرض على جيثب

تحميل دفتر

انظر نموذج TF Hub

في هذا البرنامج التعليمي، وسوف تتعلم كيفية استخدام المؤشرات الإنصاف لتقييم التضمينات من TF المحور . يستخدم هذا الكمبيوتر الدفتري و التعليقات المدني مجموعة البيانات .

يثبت

قم بتثبيت المكتبات المطلوبة.

!pip install -q -U pip==20.2

!pip install fairness-indicators \
  "absl-py==0.12.0" \
  "pyarrow==2.0.0" \
  "apache-beam==2.34.0" \
  "avro-python3==1.9.1"

استيراد المكتبات الأخرى المطلوبة.

import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model
from fairness_indicators.tutorial_utils import util

ERROR: 
Traceback (most recent call last):
  File "/tmpfs/src/tf_docs_env/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py", line 341, in <module>
    import google.cloud.bigquery_storage_v1 as bq_storage
ModuleNotFoundError: No module named 'google.cloud.bigquery_storage_v1'

مجموعة البيانات

في هذه المفكرة، التي تعمل مع التعليقات المدني مجموعة البيانات التي تحتوي على ما يقرب من 2 مليون التعليقات العامة التي نشرها المدنية التعليقات منصة في عام 2017 للبحث المستمر. تمت رعاية هذا الجهد بواسطة Jigsaw ، الذين استضافوا مسابقات على Kaggle للمساعدة في تصنيف التعليقات السامة بالإضافة إلى تقليل تحيز النموذج غير المقصود.

يحتوي كل تعليق نصي فردي في مجموعة البيانات على ملصق سمية ، بحيث يكون الملصق 1 إذا كان التعليق سامًا و 0 إذا كان التعليق غير سام. ضمن البيانات ، يتم تصنيف مجموعة فرعية من التعليقات بمجموعة متنوعة من سمات الهوية ، بما في ذلك فئات الجنس والتوجه الجنسي والدين والعرق أو العرق.

تحضير البيانات

TensorFlow يوزع الميزات من البيانات باستخدام tf.io.FixedLenFeature و tf.io.VarLenFeature . حدد ميزة الإدخال وميزة الإخراج وجميع ميزات التقطيع الأخرى ذات الأهمية.

BASE_DIR = tempfile.gettempdir()

# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'

FEATURE_MAP = {
    # input and output features
    LABEL: tf.io.FixedLenFeature([], tf.float32),
    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),

    # slicing features
    'sexual_orientation': tf.io.VarLenFeature(tf.string),
    'gender': tf.io.VarLenFeature(tf.string),
    'religion': tf.io.VarLenFeature(tf.string),
    'race': tf.io.VarLenFeature(tf.string),
    'disability': tf.io.VarLenFeature(tf.string)
}

IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']

بشكل افتراضي ، يقوم الكمبيوتر الدفتري بتنزيل إصدار مُعالج مسبقًا من مجموعة البيانات هذه ، ولكن يمكنك استخدام مجموعة البيانات الأصلية وإعادة تشغيل خطوات المعالجة إذا كنت ترغب في ذلك.

في مجموعة البيانات الأصلية ، يتم تصنيف كل تعليق بالنسبة المئوية للمقيمين الذين يعتقدون أن تعليقًا يتوافق مع هوية معينة. على سبيل المثال، قد يكون المسمى تعليق بما يلي: { male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0 } .

تقوم خطوة المعالجة بتجميع الهوية حسب الفئة (الجنس ، والتوجه الجنسي ، وما إلى ذلك) وتزيل الهويات التي تقل درجة عن 0.5. لذلك سيتم تحويل المثال أعلاه إلى ما يلي: للمقيمين الذين اعتقدوا أن تعليقًا يتوافق مع هوية معينة. على سبيل المثال، التعليق الوارد أعلاه أن يكون المسمى مع ما يلي: { gender: [female], sexual_orientation: [heterosexual, homosexual_gay_or_lesbian] }

قم بتنزيل مجموعة البيانات.

download_original_data = False

if download_original_data:
  train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')

  # The identity terms list will be grouped together by their categories
  # (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,
  # text column and label column will be kept after processing.
  train_tf_file = util.convert_comments_data(train_tf_file)
  validate_tf_file = util.convert_comments_data(validate_tf_file)

else:
  train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')

Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord
488161280/488153424 [==============================] - 2s 0us/step
488169472/488153424 [==============================] - 2s 0us/step
Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord
324943872/324941336 [==============================] - 9s 0us/step
324952064/324941336 [==============================] - 9s 0us/step

قم بإنشاء خط أنابيب تحليل نموذج TensorFlow

تعمل المكتبة مؤشرات الإنصاف على TensorFlow تحليل نموذج (TFMA) نماذج . نماذج TFMA تغلف نماذج TensorFlow بوظائف إضافية لتقييم وتصور نتائجها. يحدث التقييم الفعلي داخل من خط أنابيب أباتشي شعاع .

الخطوات التي تتبعها لإنشاء خط أنابيب TFMA هي:

1. بناء نموذج TensorFlow
2. قم ببناء نموذج TFMA أعلى نموذج TensorFlow
3. قم بتشغيل تحليل النموذج في منسق. يستخدم النموذج النموذجي في هذا الكمبيوتر الدفتري Apache Beam كمنسق.

def embedding_fairness_result(embedding, identity_term='gender'):

  model_dir = os.path.join(BASE_DIR, 'train',
                         datetime.now().strftime('%Y%m%d-%H%M%S'))

  print("Training classifier for " + embedding)
  classifier = example_model.train_model(model_dir,
                                         train_tf_file,
                                         LABEL,
                                         TEXT_FEATURE,
                                         FEATURE_MAP,
                                         embedding)

  # Create a unique path to store the results for this embedding.
  embedding_name = embedding.split('/')[-2]
  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)

  example_model.evaluate_model(classifier,
                               validate_tf_file,
                               eval_result_path,
                               identity_term,
                               LABEL,
                               FEATURE_MAP)
  return tfma.load_eval_result(output_path=eval_result_path)

قم بتشغيل مؤشرات TFMA والإنصاف

مقاييس مؤشرات الإنصاف

بعض المقاييس المتوفرة مع مؤشرات الإنصاف هي:

• المعدل السلبي والمعدل السلبي الكاذب (FNR) والمعدل السلبي الحقيقي (TNR)
• المعدل الإيجابي والمعدل الإيجابي الكاذب (FPR) والمعدل الإيجابي الحقيقي (TPR)
• دقة
• الدقة والاستدعاء
• الدقة - استدعاء AUC
• ROC AUC

حفلات الزفاف النصية

TF-محور يوفر العديد من التضمينات النص. ستكون هذه الزخارف بمثابة عمود ميزة للطرز المختلفة. يستخدم هذا البرنامج التعليمي حفلات الزفاف التالية:

• عشوائية-nnlm-أون-dim128 : التضمينات نص عشوائي، وهذا يخدم كأساس مناسب.
• nnlm-أون-dim128 : نص تضمين على أساس والعصبية الاحتمالية اللغة نموذج .
• عالميا، بالسجن لالتشفير : نص التضمين على أساس عالمي الجملة التشفير .

نتائج مؤشر الإنصاف

مؤشرات الإنصاف حساب مع embedding_fairness_result خط أنابيب، ومن ثم تقديم النتائج في الإنصاف المؤشر UI القطعة مع widget_view.render_fairness_indicator لجميع التضمينات أعلاه.

عشوائية NNLM

eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')

Training classifier for https://tfhub.dev/google/random-nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:22:54.196242: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 60.23522, step = 0
INFO:tensorflow:loss = 60.23522, step = 0
INFO:tensorflow:global_step/sec: 78.2958
INFO:tensorflow:global_step/sec: 78.2958
INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec)
INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec)
INFO:tensorflow:global_step/sec: 85.8245
INFO:tensorflow:global_step/sec: 85.8245
INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec)
INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec)
INFO:tensorflow:global_step/sec: 83.7495
INFO:tensorflow:global_step/sec: 83.7495
INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec)
INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec)
INFO:tensorflow:global_step/sec: 83.0013
INFO:tensorflow:global_step/sec: 83.0013
INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec)
INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec)
INFO:tensorflow:global_step/sec: 83.4782
INFO:tensorflow:global_step/sec: 83.4782
INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec)
INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec)
INFO:tensorflow:global_step/sec: 87.0099
INFO:tensorflow:global_step/sec: 87.0099
INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec)
INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec)
INFO:tensorflow:global_step/sec: 86.7988
INFO:tensorflow:global_step/sec: 86.7988
INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec)
INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec)
INFO:tensorflow:global_step/sec: 88.1099
INFO:tensorflow:global_step/sec: 88.1099
INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec)
INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec)
INFO:tensorflow:global_step/sec: 85.3134
INFO:tensorflow:global_step/sec: 85.3134
INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec)
INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 59.963802.
INFO:tensorflow:Loss for final step: 59.963802.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:23:11.033169: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb
WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`

widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)

FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

NNLM

eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')

Training classifier for https://tfhub.dev/google/nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:25:24.785154: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 58.637047, step = 0
INFO:tensorflow:loss = 58.637047, step = 0
INFO:tensorflow:global_step/sec: 75.6907
INFO:tensorflow:global_step/sec: 75.6907
INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec)
INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec)
INFO:tensorflow:global_step/sec: 85.4193
INFO:tensorflow:global_step/sec: 85.4193
INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec)
INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec)
INFO:tensorflow:global_step/sec: 85.3916
INFO:tensorflow:global_step/sec: 85.3916
INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec)
INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec)
INFO:tensorflow:global_step/sec: 85.7359
INFO:tensorflow:global_step/sec: 85.7359
INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec)
INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec)
INFO:tensorflow:global_step/sec: 85.6231
INFO:tensorflow:global_step/sec: 85.6231
INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec)
INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec)
INFO:tensorflow:global_step/sec: 85.1399
INFO:tensorflow:global_step/sec: 85.1399
INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec)
INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec)
INFO:tensorflow:global_step/sec: 83.6346
INFO:tensorflow:global_step/sec: 83.6346
INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec)
INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec)
INFO:tensorflow:global_step/sec: 85.4834
INFO:tensorflow:global_step/sec: 85.4834
INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec)
INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec)
INFO:tensorflow:global_step/sec: 86.7353
INFO:tensorflow:global_step/sec: 86.7353
INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec)
INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 50.57175.
INFO:tensorflow:Loss for final step: 50.57175.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:25:40.091474: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb
WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables

widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)

FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'label/mean'…

التشفير الشامل

eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')

Training classifier for https://tfhub.dev/google/universal-sentence-encoder/2
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
2022-01-07 18:28:15.955057: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.228935, step = 0
INFO:tensorflow:loss = 59.228935, step = 0
INFO:tensorflow:global_step/sec: 8.64079
INFO:tensorflow:global_step/sec: 8.64079
INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec)
INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec)
INFO:tensorflow:global_step/sec: 8.72597
INFO:tensorflow:global_step/sec: 8.72597
INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec)
INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec)
INFO:tensorflow:global_step/sec: 9.02825
INFO:tensorflow:global_step/sec: 9.02825
INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec)
INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec)
INFO:tensorflow:global_step/sec: 9.01342
INFO:tensorflow:global_step/sec: 9.01342
INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec)
INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec)
INFO:tensorflow:global_step/sec: 8.952
INFO:tensorflow:global_step/sec: 8.952
INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec)
INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec)
INFO:tensorflow:global_step/sec: 9.09908
INFO:tensorflow:global_step/sec: 9.09908
INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec)
INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec)
INFO:tensorflow:global_step/sec: 9.02127
INFO:tensorflow:global_step/sec: 9.02127
INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec)
INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec)
INFO:tensorflow:global_step/sec: 9.09376
INFO:tensorflow:global_step/sec: 9.09376
INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec)
INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec)
INFO:tensorflow:global_step/sec: 9.11679
INFO:tensorflow:global_step/sec: 9.11679
INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec)
INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 46.92047.
INFO:tensorflow:Loss for final step: 46.92047.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
2022-01-07 18:30:32.176628: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb
WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables

widget_view.render_fairness_indicator(eval_result=eval_result_use)

FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

مقارنة حفلات الزفاف

يمكنك أيضًا استخدام مؤشرات الإنصاف لمقارنة حفلات الزفاف مباشرةً. على سبيل المثال ، قارن بين النماذج التي تم إنشاؤها من حفلات الزفاف NNLM و USE.

widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})

FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…