يوم مجتمع ML هو 9 نوفمبر! الانضمام إلينا للحصول على التحديثات من TensorFlow، JAX، وأكثر معرفة المزيد

مؤشرات الإنصاف على TF-Hub Text Embeddings

عرض على TensorFlow.org تشغيل في Google Colab عرض على جيثب تحميل دفتر انظر نموذج TF Hub

في هذا البرنامج التعليمي ، ستتعلم كيفية استخدام مؤشرات الإنصاف لتقييم حفلات الزفاف من TF Hub . يستخدم هذا الكمبيوتر الدفتري مجموعة بيانات التعليقات المدنية .

يثبت

قم بتثبيت المكتبات المطلوبة.

!pip install -q -U pip==20.2

!pip install fairness-indicators \
  "absl-py==0.12.0" \
  "pyarrow==2.0.0" \
  "apache-beam==2.28.0" \
  "avro-python3==1.9.1"

استيراد المكتبات الأخرى المطلوبة.

import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model
from fairness_indicators.tutorial_utils import util

مجموعة البيانات

في هذا دفتر الملاحظات ، أنت تعمل مع مجموعة بيانات التعليقات المدنية التي تحتوي على ما يقرب من 2 مليون تعليق عام تم نشرها بواسطة منصة التعليقات المدنية في عام 2017 من أجل البحث المستمر. تمت رعاية هذا الجهد بواسطة Jigsaw ، الذين استضافوا مسابقات على Kaggle للمساعدة في تصنيف التعليقات السامة بالإضافة إلى تقليل تحيز النموذج غير المقصود.

يحتوي كل تعليق نصي فردي في مجموعة البيانات على ملصق سمية ، مع كون الملصق 1 إذا كان التعليق سامًا و 0 إذا كان التعليق غير سام. ضمن البيانات ، يتم تصنيف مجموعة فرعية من التعليقات بمجموعة متنوعة من سمات الهوية ، بما في ذلك فئات الجنس والتوجه الجنسي والدين والعرق أو العرق.

تحضير البيانات

يحلل TensorFlow الميزات من البيانات باستخدام tf.io.FixedLenFeature و tf.io.VarLenFeature . حدد ميزة الإدخال وميزة الإخراج وجميع ميزات التقطيع الأخرى ذات الأهمية.

BASE_DIR = tempfile.gettempdir()

# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'

FEATURE_MAP = {
    # input and output features
    LABEL: tf.io.FixedLenFeature([], tf.float32),
    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),

    # slicing features
    'sexual_orientation': tf.io.VarLenFeature(tf.string),
    'gender': tf.io.VarLenFeature(tf.string),
    'religion': tf.io.VarLenFeature(tf.string),
    'race': tf.io.VarLenFeature(tf.string),
    'disability': tf.io.VarLenFeature(tf.string)
}

IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']

بشكل افتراضي ، يقوم الكمبيوتر الدفتري بتنزيل إصدار مُعالج مسبقًا من مجموعة البيانات هذه ، ولكن يمكنك استخدام مجموعة البيانات الأصلية وإعادة تشغيل خطوات المعالجة إذا كنت ترغب في ذلك.

في مجموعة البيانات الأصلية ، يتم تصنيف كل تعليق بالنسبة المئوية للمقيمين الذين يعتقدون أن تعليقًا يتوافق مع هوية معينة. على سبيل المثال ، قد يتم تصنيف التعليق بما يلي: { male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0 } .

تقوم خطوة المعالجة بتجميع الهوية حسب الفئة (الجنس ، والتوجه الجنسي ، وما إلى ذلك) وتزيل الهويات التي تقل درجة عن 0.5. لذا فإن المثال أعلاه سيتم تحويله إلى ما يلي: للمقيمين الذين اعتقدوا أن تعليقًا يتوافق مع هوية معينة. على سبيل المثال ، سيتم تصنيف التعليق أعلاه بما يلي: { gender: [female], sexual_orientation: [heterosexual, homosexual_gay_or_lesbian] }

قم بتنزيل مجموعة البيانات.

download_original_data = False

if download_original_data:
  train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')

  # The identity terms list will be grouped together by their categories
  # (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,
  # text column and label column will be kept after processing.
  train_tf_file = util.convert_comments_data(train_tf_file)
  validate_tf_file = util.convert_comments_data(validate_tf_file)

else:
  train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')

قم بإنشاء خط أنابيب تحليل نموذج TensorFlow

تعمل مكتبة مؤشرات الإنصاف على نماذج تحليل نموذج TensorFlow (TFMA) . تقوم نماذج TFMA بتغليف نماذج TensorFlow بوظائف إضافية لتقييم وتصور نتائجها. يحدث التقييم الفعلي داخل خط أنابيب Apache Beam .

الخطوات التي تتبعها لإنشاء خط أنابيب TFMA هي:

  1. بناء نموذج TensorFlow
  2. قم ببناء نموذج TFMA أعلى نموذج TensorFlow
  3. قم بتشغيل تحليل النموذج في منسق. يستخدم النموذج النموذجي في هذا الكمبيوتر الدفتري Apache Beam كمنسق.
def embedding_fairness_result(embedding, identity_term='gender'):

  model_dir = os.path.join(BASE_DIR, 'train',
                         datetime.now().strftime('%Y%m%d-%H%M%S'))

  print("Training classifier for " + embedding)
  classifier = example_model.train_model(model_dir,
                                         train_tf_file,
                                         LABEL,
                                         TEXT_FEATURE,
                                         FEATURE_MAP,
                                         embedding)

  # Create a unique path to store the results for this embedding.
  embedding_name = embedding.split('/')[-2]
  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)

  example_model.evaluate_model(classifier,
                               validate_tf_file,
                               eval_result_path,
                               identity_term,
                               LABEL,
                               FEATURE_MAP)
  return tfma.load_eval_result(output_path=eval_result_path)

قم بتشغيل مؤشرات TFMA والإنصاف

مقاييس مؤشرات الإنصاف

بعض المقاييس المتوفرة مع مؤشرات الإنصاف هي:

حفلات الزفاف النصية

يوفر TF-Hub العديد من حفلات الزفاف النصية . ستكون هذه الزخارف بمثابة عمود ميزة للطرز المختلفة. يستخدم هذا البرنامج التعليمي حفلات الزفاف التالية:

نتائج مؤشر الإنصاف

احسب مؤشرات الإنصاف باستخدام خط أنابيب embedding_fairness_result ، ثم widget_view.render_fairness_indicator النتائج في أداة واجهة مستخدم مؤشر الإنصاف باستخدام widget_view.render_fairness_indicator لجميع عمليات التضمين المذكورة أعلاه.

عشوائية NNLM

eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/random-nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094321', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094321', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:402: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 60.320545, step = 0
INFO:tensorflow:loss = 60.320545, step = 0
INFO:tensorflow:global_step/sec: 24.4256
INFO:tensorflow:global_step/sec: 24.4256
INFO:tensorflow:loss = 68.23323, step = 100 (4.097 sec)
INFO:tensorflow:loss = 68.23323, step = 100 (4.097 sec)
INFO:tensorflow:global_step/sec: 26.1011
INFO:tensorflow:global_step/sec: 26.1011
INFO:tensorflow:loss = 58.783485, step = 200 (3.830 sec)
INFO:tensorflow:loss = 58.783485, step = 200 (3.830 sec)
INFO:tensorflow:global_step/sec: 25.7955
INFO:tensorflow:global_step/sec: 25.7955
INFO:tensorflow:loss = 59.22664, step = 300 (3.876 sec)
INFO:tensorflow:loss = 59.22664, step = 300 (3.876 sec)
INFO:tensorflow:global_step/sec: 26.2729
INFO:tensorflow:global_step/sec: 26.2729
INFO:tensorflow:loss = 61.110313, step = 400 (3.806 sec)
INFO:tensorflow:loss = 61.110313, step = 400 (3.806 sec)
INFO:tensorflow:global_step/sec: 25.7911
INFO:tensorflow:global_step/sec: 25.7911
INFO:tensorflow:loss = 57.710922, step = 500 (3.878 sec)
INFO:tensorflow:loss = 57.710922, step = 500 (3.878 sec)
INFO:tensorflow:global_step/sec: 25.7231
INFO:tensorflow:global_step/sec: 25.7231
INFO:tensorflow:loss = 56.67649, step = 600 (3.887 sec)
INFO:tensorflow:loss = 56.67649, step = 600 (3.887 sec)
INFO:tensorflow:global_step/sec: 26.233
INFO:tensorflow:global_step/sec: 26.233
INFO:tensorflow:loss = 60.52134, step = 700 (3.812 sec)
INFO:tensorflow:loss = 60.52134, step = 700 (3.812 sec)
INFO:tensorflow:global_step/sec: 26.0555
INFO:tensorflow:global_step/sec: 26.0555
INFO:tensorflow:loss = 58.405262, step = 800 (3.839 sec)
INFO:tensorflow:loss = 58.405262, step = 800 (3.839 sec)
INFO:tensorflow:global_step/sec: 24.9941
INFO:tensorflow:global_step/sec: 24.9941
INFO:tensorflow:loss = 56.718044, step = 900 (4.000 sec)
INFO:tensorflow:loss = 56.718044, step = 900 (4.000 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094321/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 59.395683.
INFO:tensorflow:Loss for final step: 59.395683.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/canned/head.py:642: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094321/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094321/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961453/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961453/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961453/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961453/saved_model.pb
WARNING:absl:Tensorflow version (2.4.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961453/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961453/variables/variables
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

NNLM

eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094614', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094614', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.234177, step = 0
INFO:tensorflow:loss = 59.234177, step = 0
INFO:tensorflow:global_step/sec: 23.9624
INFO:tensorflow:global_step/sec: 23.9624
INFO:tensorflow:loss = 55.79068, step = 100 (4.175 sec)
INFO:tensorflow:loss = 55.79068, step = 100 (4.175 sec)
INFO:tensorflow:global_step/sec: 24.7819
INFO:tensorflow:global_step/sec: 24.7819
INFO:tensorflow:loss = 47.50544, step = 200 (4.035 sec)
INFO:tensorflow:loss = 47.50544, step = 200 (4.035 sec)
INFO:tensorflow:global_step/sec: 22.4029
INFO:tensorflow:global_step/sec: 22.4029
INFO:tensorflow:loss = 55.776627, step = 300 (4.464 sec)
INFO:tensorflow:loss = 55.776627, step = 300 (4.464 sec)
INFO:tensorflow:global_step/sec: 21.9616
INFO:tensorflow:global_step/sec: 21.9616
INFO:tensorflow:loss = 55.773647, step = 400 (4.553 sec)
INFO:tensorflow:loss = 55.773647, step = 400 (4.553 sec)
INFO:tensorflow:global_step/sec: 22.3908
INFO:tensorflow:global_step/sec: 22.3908
INFO:tensorflow:loss = 41.876427, step = 500 (4.466 sec)
INFO:tensorflow:loss = 41.876427, step = 500 (4.466 sec)
INFO:tensorflow:global_step/sec: 24.1889
INFO:tensorflow:global_step/sec: 24.1889
INFO:tensorflow:loss = 45.470932, step = 600 (4.134 sec)
INFO:tensorflow:loss = 45.470932, step = 600 (4.134 sec)
INFO:tensorflow:global_step/sec: 24.5477
INFO:tensorflow:global_step/sec: 24.5477
INFO:tensorflow:loss = 51.15878, step = 700 (4.073 sec)
INFO:tensorflow:loss = 51.15878, step = 700 (4.073 sec)
INFO:tensorflow:global_step/sec: 23.6935
INFO:tensorflow:global_step/sec: 23.6935
INFO:tensorflow:loss = 47.61961, step = 800 (4.221 sec)
INFO:tensorflow:loss = 47.61961, step = 800 (4.221 sec)
INFO:tensorflow:global_step/sec: 23.1377
INFO:tensorflow:global_step/sec: 23.1377
INFO:tensorflow:loss = 48.06709, step = 900 (4.322 sec)
INFO:tensorflow:loss = 48.06709, step = 900 (4.322 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094614/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 51.064987.
INFO:tensorflow:Loss for final step: 51.064987.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094614/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094614/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961619/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1617961619/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961619/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961619/saved_model.pb
WARNING:absl:Tensorflow version (2.4.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961619/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961619/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

التشفير الشامل

eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')
Training classifier for https://tfhub.dev/google/universal-sentence-encoder/2
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094902', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210409-094902', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.33705, step = 0
INFO:tensorflow:loss = 59.33705, step = 0
INFO:tensorflow:global_step/sec: 9.17094
INFO:tensorflow:global_step/sec: 9.17094
INFO:tensorflow:loss = 49.904232, step = 100 (10.906 sec)
INFO:tensorflow:loss = 49.904232, step = 100 (10.906 sec)
INFO:tensorflow:global_step/sec: 9.5816
INFO:tensorflow:global_step/sec: 9.5816
INFO:tensorflow:loss = 45.914093, step = 200 (10.437 sec)
INFO:tensorflow:loss = 45.914093, step = 200 (10.437 sec)
INFO:tensorflow:global_step/sec: 9.42711
INFO:tensorflow:global_step/sec: 9.42711
INFO:tensorflow:loss = 48.298733, step = 300 (10.608 sec)
INFO:tensorflow:loss = 48.298733, step = 300 (10.608 sec)
INFO:tensorflow:global_step/sec: 9.5081
INFO:tensorflow:global_step/sec: 9.5081
INFO:tensorflow:loss = 44.76107, step = 400 (10.517 sec)
INFO:tensorflow:loss = 44.76107, step = 400 (10.517 sec)
INFO:tensorflow:global_step/sec: 9.58888
INFO:tensorflow:global_step/sec: 9.58888
INFO:tensorflow:loss = 35.319324, step = 500 (10.429 sec)
INFO:tensorflow:loss = 35.319324, step = 500 (10.429 sec)
INFO:tensorflow:global_step/sec: 9.51424
INFO:tensorflow:global_step/sec: 9.51424
INFO:tensorflow:loss = 41.977276, step = 600 (10.511 sec)
INFO:tensorflow:loss = 41.977276, step = 600 (10.511 sec)
INFO:tensorflow:global_step/sec: 9.55503
INFO:tensorflow:global_step/sec: 9.55503
INFO:tensorflow:loss = 40.810833, step = 700 (10.465 sec)
INFO:tensorflow:loss = 40.810833, step = 700 (10.465 sec)
INFO:tensorflow:global_step/sec: 9.57973
INFO:tensorflow:global_step/sec: 9.57973
INFO:tensorflow:loss = 37.127766, step = 800 (10.439 sec)
INFO:tensorflow:loss = 37.127766, step = 800 (10.439 sec)
INFO:tensorflow:global_step/sec: 9.51487
INFO:tensorflow:global_step/sec: 9.51487
INFO:tensorflow:loss = 32.48211, step = 900 (10.510 sec)
INFO:tensorflow:loss = 32.48211, step = 900 (10.510 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210409-094902/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 46.977016.
INFO:tensorflow:Loss for final step: 46.977016.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094902/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20210409-094902/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961882/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1617961882/saved_model.pb
WARNING:absl:Tensorflow version (2.4.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'>
WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'>
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961882/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1617961882/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_use)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

مقارنة حفلات الزفاف

يمكنك أيضًا استخدام مؤشرات الإنصاف لمقارنة حفلات الزفاف مباشرةً. على سبيل المثال ، قارن بين النماذج التي تم إنشاؤها من حفلات الزفاف NNLM و USE.

widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})
FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…