Lihat di TensorFlow.org | Jalankan di Google Colab | Lihat di GitHub | Unduh buku catatan | Lihat model TF Hub |
Dalam tutorial ini, Anda akan belajar bagaimana menggunakan Keadilan Indikator untuk mengevaluasi embeddings dari TF Hub . Notebook ini menggunakan Sipil Komentar dataset .
Mempersiapkan
Instal perpustakaan yang diperlukan.
!pip install -q -U pip==20.2
!pip install fairness-indicators \
"absl-py==0.12.0" \
"pyarrow==2.0.0" \
"apache-beam==2.34.0" \
"avro-python3==1.9.1"
Impor perpustakaan lain yang diperlukan.
import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model
from fairness_indicators.tutorial_utils import util
ERROR: Traceback (most recent call last): File "/tmpfs/src/tf_docs_env/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py", line 341, in <module> import google.cloud.bigquery_storage_v1 as bq_storage ModuleNotFoundError: No module named 'google.cloud.bigquery_storage_v1'
Himpunan data
Dalam notebook ini, Anda bekerja dengan Sipil Komentar dataset yang berisi sekitar 2 juta komentar publik dipublikasikan oleh Sipil Komentar platform yang pada tahun 2017 untuk penelitian yang sedang berlangsung. Upaya ini disponsori oleh Jigsaw, yang telah menyelenggarakan kompetisi di Kaggle untuk membantu mengklasifikasikan komentar beracun serta meminimalkan bias model yang tidak diinginkan.
Setiap komentar teks individu dalam kumpulan data memiliki label toksisitas, dengan label 1 jika komentar beracun dan 0 jika komentar tidak beracun. Di dalam data, sebagian komentar diberi label dengan berbagai atribut identitas, termasuk kategori untuk jenis kelamin, orientasi seksual, agama, dan ras atau etnis.
Siapkan datanya
TensorFlow mem-parsing fitur dari data menggunakan tf.io.FixedLenFeature
dan tf.io.VarLenFeature
. Petakan fitur input, fitur output, dan semua fitur slicing lainnya yang menarik.
BASE_DIR = tempfile.gettempdir()
# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'
FEATURE_MAP = {
# input and output features
LABEL: tf.io.FixedLenFeature([], tf.float32),
TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),
# slicing features
'sexual_orientation': tf.io.VarLenFeature(tf.string),
'gender': tf.io.VarLenFeature(tf.string),
'religion': tf.io.VarLenFeature(tf.string),
'race': tf.io.VarLenFeature(tf.string),
'disability': tf.io.VarLenFeature(tf.string)
}
IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']
Secara default, buku catatan mengunduh versi praproses dari kumpulan data ini, tetapi Anda dapat menggunakan kumpulan data asli dan menjalankan kembali langkah-langkah pemrosesan jika diinginkan.
Dalam kumpulan data asli, setiap komentar diberi label dengan persentase penilai yang percaya bahwa komentar sesuai dengan identitas tertentu. Misalnya, komentar mungkin diberi label dengan berikut ini: { male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0 }
.
Langkah pemrosesan mengelompokkan identitas berdasarkan kategori (gender, orientasi_seksual, dll.) dan menghilangkan identitas dengan skor kurang dari 0,5. Jadi contoh di atas akan dikonversi menjadi berikut: penilai yang percaya bahwa komentar sesuai dengan identitas tertentu. Sebagai contoh, komentar di atas akan diberi label dengan berikut ini: { gender: [female], sexual_orientation: [heterosexual, homosexual_gay_or_lesbian] }
Unduh kumpulan datanya.
download_original_data = False
if download_original_data:
train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')
# The identity terms list will be grouped together by their categories
# (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,
# text column and label column will be kept after processing.
train_tf_file = util.convert_comments_data(train_tf_file)
validate_tf_file = util.convert_comments_data(validate_tf_file)
else:
train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord 488161280/488153424 [==============================] - 2s 0us/step 488169472/488153424 [==============================] - 2s 0us/step Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord 324943872/324941336 [==============================] - 9s 0us/step 324952064/324941336 [==============================] - 9s 0us/step
Buat Pipeline Analisis Model TensorFlow
Kewajaran Indikator perpustakaan beroperasi pada Analisis Model TensorFlow (TFMA) model . Model TFMA membungkus model TensorFlow dengan fungsionalitas tambahan untuk mengevaluasi dan memvisualisasikan hasilnya. Evaluasi yang sebenarnya terjadi di dalam sebuah pipa Apache Beam .
Langkah-langkah yang Anda ikuti untuk membuat pipa TFMA adalah:
- Buat model TensorFlow
- Bangun model TFMA di atas model TensorFlow
- Jalankan analisis model dalam orkestra. Contoh model pada notebook ini menggunakan Apache Beam sebagai orkestra.
def embedding_fairness_result(embedding, identity_term='gender'):
model_dir = os.path.join(BASE_DIR, 'train',
datetime.now().strftime('%Y%m%d-%H%M%S'))
print("Training classifier for " + embedding)
classifier = example_model.train_model(model_dir,
train_tf_file,
LABEL,
TEXT_FEATURE,
FEATURE_MAP,
embedding)
# Create a unique path to store the results for this embedding.
embedding_name = embedding.split('/')[-2]
eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)
example_model.evaluate_model(classifier,
validate_tf_file,
eval_result_path,
identity_term,
LABEL,
FEATURE_MAP)
return tfma.load_eval_result(output_path=eval_result_path)
Jalankan TFMA & Indikator Kewajaran
Metrik Indikator Kewajaran
Beberapa metrik yang tersedia dengan Indikator Kewajaran adalah:
- Tingkat Negatif, Tingkat Negatif Palsu (FNR), dan Tingkat Negatif Benar (TNR)
- Tingkat Positif, Tingkat Positif Palsu (FPR), dan Tingkat Positif Benar (TPR)
- Ketepatan
- Presisi dan Recall
- Presisi-Recall AUC
- ROC AUC
Penyematan Teks
TF-Hub menyediakan beberapa embeddings teks. Penyematan ini akan berfungsi sebagai kolom fitur untuk model yang berbeda. Tutorial ini menggunakan embeddings berikut:
- acak-nnlm-en-dim128 : embeddings teks acak, servis ini sebagai dasar nyaman.
- nnlm-en-dim128 : teks embedding berdasarkan A Neural probabilistik Bahasa Model .
- yang universal-kalimat-encoder : teks embedding berdasarkan Universal Kalimat Encoder .
Hasil Indikator Kewajaran
Indikator keadilan menghitung dengan embedding_fairness_result
pipa, dan kemudian membuat hasil dalam Fairness Indikator UI widget dengan widget_view.render_fairness_indicator
untuk semua embeddings di atas.
NNLM Acak
eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/random-nnlm-en-dim128/1 INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:22:54.196242: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 60.23522, step = 0 INFO:tensorflow:loss = 60.23522, step = 0 INFO:tensorflow:global_step/sec: 78.2958 INFO:tensorflow:global_step/sec: 78.2958 INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec) INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec) INFO:tensorflow:global_step/sec: 85.8245 INFO:tensorflow:global_step/sec: 85.8245 INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec) INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec) INFO:tensorflow:global_step/sec: 83.7495 INFO:tensorflow:global_step/sec: 83.7495 INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec) INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec) INFO:tensorflow:global_step/sec: 83.0013 INFO:tensorflow:global_step/sec: 83.0013 INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec) INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec) INFO:tensorflow:global_step/sec: 83.4782 INFO:tensorflow:global_step/sec: 83.4782 INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec) INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec) INFO:tensorflow:global_step/sec: 87.0099 INFO:tensorflow:global_step/sec: 87.0099 INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec) INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec) INFO:tensorflow:global_step/sec: 86.7988 INFO:tensorflow:global_step/sec: 86.7988 INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec) INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec) INFO:tensorflow:global_step/sec: 88.1099 INFO:tensorflow:global_step/sec: 88.1099 INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec) INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec) INFO:tensorflow:global_step/sec: 85.3134 INFO:tensorflow:global_step/sec: 85.3134 INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec) INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 59.963802. INFO:tensorflow:Loss for final step: 59.963802. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:23:11.033169: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…
NNLM
eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/nnlm-en-dim128/1 INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:25:24.785154: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 58.637047, step = 0 INFO:tensorflow:loss = 58.637047, step = 0 INFO:tensorflow:global_step/sec: 75.6907 INFO:tensorflow:global_step/sec: 75.6907 INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec) INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec) INFO:tensorflow:global_step/sec: 85.4193 INFO:tensorflow:global_step/sec: 85.4193 INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec) INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec) INFO:tensorflow:global_step/sec: 85.3916 INFO:tensorflow:global_step/sec: 85.3916 INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec) INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec) INFO:tensorflow:global_step/sec: 85.7359 INFO:tensorflow:global_step/sec: 85.7359 INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec) INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec) INFO:tensorflow:global_step/sec: 85.6231 INFO:tensorflow:global_step/sec: 85.6231 INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec) INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec) INFO:tensorflow:global_step/sec: 85.1399 INFO:tensorflow:global_step/sec: 85.1399 INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec) INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec) INFO:tensorflow:global_step/sec: 83.6346 INFO:tensorflow:global_step/sec: 83.6346 INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec) INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec) INFO:tensorflow:global_step/sec: 85.4834 INFO:tensorflow:global_step/sec: 85.4834 INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec) INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec) INFO:tensorflow:global_step/sec: 86.7353 INFO:tensorflow:global_step/sec: 86.7353 INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec) INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 50.57175. INFO:tensorflow:Loss for final step: 50.57175. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:25:40.091474: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'label/mean'…
Encoder Kalimat Universal
eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')
Training classifier for https://tfhub.dev/google/universal-sentence-encoder/2 INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. 2022-01-07 18:28:15.955057: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 59.228935, step = 0 INFO:tensorflow:loss = 59.228935, step = 0 INFO:tensorflow:global_step/sec: 8.64079 INFO:tensorflow:global_step/sec: 8.64079 INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec) INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec) INFO:tensorflow:global_step/sec: 8.72597 INFO:tensorflow:global_step/sec: 8.72597 INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec) INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec) INFO:tensorflow:global_step/sec: 9.02825 INFO:tensorflow:global_step/sec: 9.02825 INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec) INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec) INFO:tensorflow:global_step/sec: 9.01342 INFO:tensorflow:global_step/sec: 9.01342 INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec) INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec) INFO:tensorflow:global_step/sec: 8.952 INFO:tensorflow:global_step/sec: 8.952 INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec) INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec) INFO:tensorflow:global_step/sec: 9.09908 INFO:tensorflow:global_step/sec: 9.09908 INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec) INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec) INFO:tensorflow:global_step/sec: 9.02127 INFO:tensorflow:global_step/sec: 9.02127 INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec) INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec) INFO:tensorflow:global_step/sec: 9.09376 INFO:tensorflow:global_step/sec: 9.09376 INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec) INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec) INFO:tensorflow:global_step/sec: 9.11679 INFO:tensorflow:global_step/sec: 9.11679 INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec) INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 46.92047. INFO:tensorflow:Loss for final step: 46.92047. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. 2022-01-07 18:30:32.176628: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:No assets to write. INFO:tensorflow:No assets to write. INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_use)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…
Membandingkan Penyematan
Anda juga dapat menggunakan Indikator Kewajaran untuk membandingkan penyematan secara langsung. Misalnya, bandingkan model yang dihasilkan dari penyematan NNLM dan USE.
widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})
FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…