TensorFlow.orgで表示 | GoogleColabで実行 | GitHubで表示 | ノートブックをダウンロード | TFハブモデルを参照してください |
このチュートリアルでは、使用する方法を学びますフェアネスインジケータから埋め込み評価するTFハブ。このノートPCは使って市民がデータセットをコメント。
設定
必要なライブラリをインストールします。
!pip install -q -U pip==20.2
!pip install fairness-indicators \
"absl-py==0.12.0" \
"pyarrow==2.0.0" \
"apache-beam==2.34.0" \
"avro-python3==1.9.1"
他の必要なライブラリをインポートします。
import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model
from fairness_indicators.tutorial_utils import util
ERROR: Traceback (most recent call last): File "/tmpfs/src/tf_docs_env/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py", line 341, in <module> import google.cloud.bigquery_storage_v1 as bq_storage ModuleNotFoundError: No module named 'google.cloud.bigquery_storage_v1'
データセット
このノートブックでは、あなたが働く市民がデータセットコメントで公開されて約200万パブリックコメントが含まれている市民をプラットフォームコメント進行中の研究のために2017年に。この取り組みは、有毒なコメントを分類し、意図しないモデルのバイアスを最小限に抑えるために、Kaggleでコンテストを主催したJigsawによって後援されました。
データセット内の個々のテキストコメントには毒性ラベルがあり、コメントが毒性の場合は1、コメントが非毒性の場合は0になります。データ内で、コメントのサブセットは、性別、性的指向、宗教、人種または民族のカテゴリなど、さまざまなID属性でラベル付けされています。
データを準備する
TensorFlowは、使用したデータから特徴を解析しtf.io.FixedLenFeature
とtf.io.VarLenFeature
。入力機能、出力機能、およびその他すべての対象のスライス機能をマップします。
BASE_DIR = tempfile.gettempdir()
# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'
FEATURE_MAP = {
# input and output features
LABEL: tf.io.FixedLenFeature([], tf.float32),
TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),
# slicing features
'sexual_orientation': tf.io.VarLenFeature(tf.string),
'gender': tf.io.VarLenFeature(tf.string),
'religion': tf.io.VarLenFeature(tf.string),
'race': tf.io.VarLenFeature(tf.string),
'disability': tf.io.VarLenFeature(tf.string)
}
IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']
デフォルトでは、ノートブックはこのデータセットの前処理されたバージョンをダウンロードしますが、元のデータセットを使用して、必要に応じて処理ステップを再実行することもできます。
元のデータセットでは、各コメントには、コメントが特定のIDに対応すると信じている評価者の割合がラベル付けされています。例えば、コメントは以下で標識されることがあります: { male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0 }
処理ステップでは、IDをカテゴリ(性別、性的指向など)ごとにグループ化し、スコアが0.5未満のIDを削除します。したがって、上記の例は次のように変換されます。コメントが特定のアイデンティティに対応すると信じている評価者の場合。例えば、上記のコメントは、以下で標識されるであろう: { gender: [female], sexual_orientation: [heterosexual, homosexual_gay_or_lesbian] }
データセットをダウンロードします。
download_original_data = False
if download_original_data:
train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')
# The identity terms list will be grouped together by their categories
# (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,
# text column and label column will be kept after processing.
train_tf_file = util.convert_comments_data(train_tf_file)
validate_tf_file = util.convert_comments_data(validate_tf_file)
else:
train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord 488161280/488153424 [==============================] - 2s 0us/step 488169472/488153424 [==============================] - 2s 0us/step Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord 324943872/324941336 [==============================] - 9s 0us/step 324952064/324941336 [==============================] - 9s 0us/step
TensorFlowモデル分析パイプラインを作成する
公正指標ライブラリは、上で動作しTensorFlowモデル分析(TFMA)モデル。 TFMAモデルは、結果を評価および視覚化するための追加機能を備えたTensorFlowモデルをラップします。実際の評価は、内部で発生したApacheのビームパイプライン。
TFMAパイプラインを作成するために従う手順は次のとおりです。
- TensorFlowモデルを構築する
- TensorFlowモデルの上にTFMAモデルを構築する
- オーケストレーターでモデル分析を実行します。このノートブックのサンプルモデルは、オーケストレーターとしてApacheBeamを使用しています。
def embedding_fairness_result(embedding, identity_term='gender'):
model_dir = os.path.join(BASE_DIR, 'train',
datetime.now().strftime('%Y%m%d-%H%M%S'))
print("Training classifier for " + embedding)
classifier = example_model.train_model(model_dir,
train_tf_file,
LABEL,
TEXT_FEATURE,
FEATURE_MAP,
embedding)
# Create a unique path to store the results for this embedding.
embedding_name = embedding.split('/')[-2]
eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)
example_model.evaluate_model(classifier,
validate_tf_file,
eval_result_path,
identity_term,
LABEL,
FEATURE_MAP)
return tfma.load_eval_result(output_path=eval_result_path)
TFMAと公平性指標を実行する
公平性指標の指標
公平性指標で利用可能なメトリックのいくつかは次のとおりです。
テキストの埋め込み
TF-Hubは、いくつかのテキスト埋め込みを提供します。これらの埋め込みは、さまざまなモデルの機能列として機能します。このチュートリアルでは、次の埋め込みを使用します。
- ランダムnnlmエンdim128 :ランダムテキスト埋め込み、これは便利なベースラインとして機能します。
- nnlmエンdim128 :テキストに基づいて埋め込むAニューラル確率的言語モデル。
- ユニバーサル・文・エンコーダ:をもとに埋め込むテキストユニバーサル文エンコーダ。
公平性指標の結果
計算の公平性を持つ指標embedding_fairness_result
パイプライン、そして公正インジケータUIで結果をレンダリングするには、ウィジェットとwidget_view.render_fairness_indicator
上記のすべての埋め込みのために。
ランダムNNLM
eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/random-nnlm-en-dim128/1 INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:22:54.196242: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 60.23522, step = 0 INFO:tensorflow:loss = 60.23522, step = 0 INFO:tensorflow:global_step/sec: 78.2958 INFO:tensorflow:global_step/sec: 78.2958 INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec) INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec) INFO:tensorflow:global_step/sec: 85.8245 INFO:tensorflow:global_step/sec: 85.8245 INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec) INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec) INFO:tensorflow:global_step/sec: 83.7495 INFO:tensorflow:global_step/sec: 83.7495 INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec) INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec) INFO:tensorflow:global_step/sec: 83.0013 INFO:tensorflow:global_step/sec: 83.0013 INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec) INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec) INFO:tensorflow:global_step/sec: 83.4782 INFO:tensorflow:global_step/sec: 83.4782 INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec) INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec) INFO:tensorflow:global_step/sec: 87.0099 INFO:tensorflow:global_step/sec: 87.0099 INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec) INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec) INFO:tensorflow:global_step/sec: 86.7988 INFO:tensorflow:global_step/sec: 86.7988 INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec) INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec) INFO:tensorflow:global_step/sec: 88.1099 INFO:tensorflow:global_step/sec: 88.1099 INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec) INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec) INFO:tensorflow:global_step/sec: 85.3134 INFO:tensorflow:global_step/sec: 85.3134 INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec) INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 59.963802. INFO:tensorflow:Loss for final step: 59.963802. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:23:11.033169: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…
NNLM
eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/nnlm-en-dim128/1 INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:25:24.785154: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 58.637047, step = 0 INFO:tensorflow:loss = 58.637047, step = 0 INFO:tensorflow:global_step/sec: 75.6907 INFO:tensorflow:global_step/sec: 75.6907 INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec) INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec) INFO:tensorflow:global_step/sec: 85.4193 INFO:tensorflow:global_step/sec: 85.4193 INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec) INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec) INFO:tensorflow:global_step/sec: 85.3916 INFO:tensorflow:global_step/sec: 85.3916 INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec) INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec) INFO:tensorflow:global_step/sec: 85.7359 INFO:tensorflow:global_step/sec: 85.7359 INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec) INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec) INFO:tensorflow:global_step/sec: 85.6231 INFO:tensorflow:global_step/sec: 85.6231 INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec) INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec) INFO:tensorflow:global_step/sec: 85.1399 INFO:tensorflow:global_step/sec: 85.1399 INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec) INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec) INFO:tensorflow:global_step/sec: 83.6346 INFO:tensorflow:global_step/sec: 83.6346 INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec) INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec) INFO:tensorflow:global_step/sec: 85.4834 INFO:tensorflow:global_step/sec: 85.4834 INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec) INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec) INFO:tensorflow:global_step/sec: 86.7353 INFO:tensorflow:global_step/sec: 86.7353 INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec) INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 50.57175. INFO:tensorflow:Loss for final step: 50.57175. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore 2022-01-07 18:25:40.091474: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'label/mean'…
ユニバーサルセンテンスエンコーダ
eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')
Training classifier for https://tfhub.dev/google/universal-sentence-encoder/2 INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. 2022-01-07 18:28:15.955057: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 59.228935, step = 0 INFO:tensorflow:loss = 59.228935, step = 0 INFO:tensorflow:global_step/sec: 8.64079 INFO:tensorflow:global_step/sec: 8.64079 INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec) INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec) INFO:tensorflow:global_step/sec: 8.72597 INFO:tensorflow:global_step/sec: 8.72597 INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec) INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec) INFO:tensorflow:global_step/sec: 9.02825 INFO:tensorflow:global_step/sec: 9.02825 INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec) INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec) INFO:tensorflow:global_step/sec: 9.01342 INFO:tensorflow:global_step/sec: 9.01342 INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec) INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec) INFO:tensorflow:global_step/sec: 8.952 INFO:tensorflow:global_step/sec: 8.952 INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec) INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec) INFO:tensorflow:global_step/sec: 9.09908 INFO:tensorflow:global_step/sec: 9.09908 INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec) INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec) INFO:tensorflow:global_step/sec: 9.02127 INFO:tensorflow:global_step/sec: 9.02127 INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec) INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec) INFO:tensorflow:global_step/sec: 9.09376 INFO:tensorflow:global_step/sec: 9.09376 INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec) INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec) INFO:tensorflow:global_step/sec: 9.11679 INFO:tensorflow:global_step/sec: 9.11679 INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec) INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 46.92047. INFO:tensorflow:Loss for final step: 46.92047. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. 2022-01-07 18:30:32.176628: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:No assets to write. INFO:tensorflow:No assets to write. INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_use)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…
埋め込みの比較
公平性インジケーターを使用して、埋め込みを直接比較することもできます。たとえば、NNLM埋め込みとUSE埋め込みから生成されたモデルを比較します。
widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})
FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…