Indicadores de imparcialidade em incorporações de texto TF-Hub

Ver no TensorFlow.org Executar no Google Colab Ver no GitHub Baixar caderno Veja o modelo TF Hub

Neste tutorial, você vai aprender como usar Fairness Indicadores para avaliar embeddings de TF Hub . Este notebook usa o Civil Comentários dataset .

Configurar

Instale as bibliotecas necessárias.

!pip install -q -U pip==20.2

!pip install fairness-indicators \
  "absl-py==0.12.0" \
  "pyarrow==2.0.0" \
  "apache-beam==2.34.0" \
  "avro-python3==1.9.1"

Importe outras bibliotecas necessárias.

import os
import tempfile
import apache_beam as beam
from datetime import datetime
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_model_analysis as tfma
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from fairness_indicators import example_model
from fairness_indicators.tutorial_utils import util
ERROR: 
Traceback (most recent call last):
  File "/tmpfs/src/tf_docs_env/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py", line 341, in <module>
    import google.cloud.bigquery_storage_v1 as bq_storage
ModuleNotFoundError: No module named 'google.cloud.bigquery_storage_v1'

Conjunto de dados

Neste caderno, você trabalha com o Civil Comentários conjunto de dados que contém aproximadamente 2 milhões de comentários públicos tornados públicos pelo Civil Comentários plataforma em 2017 para investigação em curso. Este esforço foi patrocinado pela Jigsaw, que hospedou competições no Kaggle para ajudar a classificar comentários tóxicos, bem como minimizar o preconceito involuntário do modelo.

Cada comentário de texto individual no conjunto de dados tem um rótulo de toxicidade, com o rótulo sendo 1 se o comentário for tóxico e 0 se o comentário for atóxico. Nos dados, um subconjunto de comentários é rotulado com uma variedade de atributos de identidade, incluindo categorias de gênero, orientação sexual, religião e raça ou etnia.

Prepare os dados

TensorFlow analisa recursos de dados usando tf.io.FixedLenFeature e tf.io.VarLenFeature . Mapeie o recurso de entrada, o recurso de saída e todos os outros recursos de fatiamento de interesse.

BASE_DIR = tempfile.gettempdir()

# The input and output features of the classifier
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'

FEATURE_MAP = {
    # input and output features
    LABEL: tf.io.FixedLenFeature([], tf.float32),
    TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),

    # slicing features
    'sexual_orientation': tf.io.VarLenFeature(tf.string),
    'gender': tf.io.VarLenFeature(tf.string),
    'religion': tf.io.VarLenFeature(tf.string),
    'race': tf.io.VarLenFeature(tf.string),
    'disability': tf.io.VarLenFeature(tf.string)
}

IDENTITY_TERMS = ['gender', 'sexual_orientation', 'race', 'religion', 'disability']

Por padrão, o notebook baixa uma versão pré-processada deste conjunto de dados, mas você pode usar o conjunto de dados original e executar novamente as etapas de processamento, se desejar.

No conjunto de dados original, cada comentário é rotulado com a porcentagem de avaliadores que acreditaram que um comentário corresponde a uma identidade específica. Por exemplo, um comentário pode ser rotulado com o seguinte: { male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0 } .

A etapa de processamento agrupa a identidade por categoria (gênero, sexual_orientation etc.) e remove as identidades com uma pontuação inferior a 0,5. Portanto, o exemplo acima seria convertido no seguinte: de avaliadores que acreditam que um comentário corresponde a uma identidade específica. Por exemplo, o comentário acima seria rotulado com o seguinte: { gender: [female], sexual_orientation: [heterosexual, homosexual_gay_or_lesbian] }

Baixe o conjunto de dados.

download_original_data = False

if download_original_data:
  train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')

  # The identity terms list will be grouped together by their categories
  # (see 'IDENTITY_COLUMNS') on threshold 0.5. Only the identity term column,
  # text column and label column will be kept after processing.
  train_tf_file = util.convert_comments_data(train_tf_file)
  validate_tf_file = util.convert_comments_data(validate_tf_file)

else:
  train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
                                          'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
  validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
                                             'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord
488161280/488153424 [==============================] - 2s 0us/step
488169472/488153424 [==============================] - 2s 0us/step
Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord
324943872/324941336 [==============================] - 9s 0us/step
324952064/324941336 [==============================] - 9s 0us/step

Crie um pipeline de análise de modelo do TensorFlow

A biblioteca Fairness Indicadores opera em TensorFlow Modelo de Análise (TFMA) modelos . Os modelos TFMA envolvem os modelos do TensorFlow com funcionalidades adicionais para avaliar e visualizar seus resultados. A avaliação propriamente dita ocorre no interior de um oleoduto Apache feixe .

As etapas que você segue para criar um pipeline TFMA são:

  1. Construir um modelo TensorFlow
  2. Construir um modelo TFMA com base no modelo TensorFlow
  3. Execute a análise do modelo em um orquestrador. O modelo de exemplo neste bloco de notas usa o Apache Beam como orquestrador.
def embedding_fairness_result(embedding, identity_term='gender'):

  model_dir = os.path.join(BASE_DIR, 'train',
                         datetime.now().strftime('%Y%m%d-%H%M%S'))

  print("Training classifier for " + embedding)
  classifier = example_model.train_model(model_dir,
                                         train_tf_file,
                                         LABEL,
                                         TEXT_FEATURE,
                                         FEATURE_MAP,
                                         embedding)

  # Create a unique path to store the results for this embedding.
  embedding_name = embedding.split('/')[-2]
  eval_result_path = os.path.join(BASE_DIR, 'eval_result', embedding_name)

  example_model.evaluate_model(classifier,
                               validate_tf_file,
                               eval_result_path,
                               identity_term,
                               LABEL,
                               FEATURE_MAP)
  return tfma.load_eval_result(output_path=eval_result_path)

Executar TFMA e indicadores de justiça

Métricas de indicadores de justiça

Algumas das métricas disponíveis com os indicadores de imparcialidade são:

Embeddings de texto

TF-Hub fornece várias incorporações de texto. Esses embeddings servirão como coluna de recursos para os diferentes modelos. Este tutorial usa os seguintes embeddings:

Resultados do indicador de justiça

Indicadores de equidade de computação com o embedding_fairness_result gasoduto, e depois processar os resultados do Indicador UI Fairness widget com widget_view.render_fairness_indicator para todos os mergulhos anteriores.

NNLM aleatório

eval_result_random_nnlm = embedding_fairness_result('https://tfhub.dev/google/random-nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/random-nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182244', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:397: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:22:54.196242: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:400: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/feature_column/feature_column.py:2188: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow/python/training/adagrad.py:139: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 60.23522, step = 0
INFO:tensorflow:loss = 60.23522, step = 0
INFO:tensorflow:global_step/sec: 78.2958
INFO:tensorflow:global_step/sec: 78.2958
INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec)
INFO:tensorflow:loss = 67.36491, step = 100 (1.279 sec)
INFO:tensorflow:global_step/sec: 85.8245
INFO:tensorflow:global_step/sec: 85.8245
INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec)
INFO:tensorflow:loss = 57.875557, step = 200 (1.165 sec)
INFO:tensorflow:global_step/sec: 83.7495
INFO:tensorflow:global_step/sec: 83.7495
INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec)
INFO:tensorflow:loss = 61.091763, step = 300 (1.194 sec)
INFO:tensorflow:global_step/sec: 83.0013
INFO:tensorflow:global_step/sec: 83.0013
INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec)
INFO:tensorflow:loss = 62.251183, step = 400 (1.205 sec)
INFO:tensorflow:global_step/sec: 83.4782
INFO:tensorflow:global_step/sec: 83.4782
INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec)
INFO:tensorflow:loss = 56.21132, step = 500 (1.198 sec)
INFO:tensorflow:global_step/sec: 87.0099
INFO:tensorflow:global_step/sec: 87.0099
INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec)
INFO:tensorflow:loss = 57.211937, step = 600 (1.149 sec)
INFO:tensorflow:global_step/sec: 86.7988
INFO:tensorflow:global_step/sec: 86.7988
INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec)
INFO:tensorflow:loss = 62.16255, step = 700 (1.152 sec)
INFO:tensorflow:global_step/sec: 88.1099
INFO:tensorflow:global_step/sec: 88.1099
INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec)
INFO:tensorflow:loss = 58.081688, step = 800 (1.135 sec)
INFO:tensorflow:global_step/sec: 85.3134
INFO:tensorflow:global_step/sec: 85.3134
INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec)
INFO:tensorflow:loss = 57.763985, step = 900 (1.172 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182244/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 59.963802.
INFO:tensorflow:Loss for final step: 59.963802.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:132: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:23:11.033169: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/canned/head.py:640: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version.
Instructions for updating:
The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182244/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579790/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579790/saved_model.pb
WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:164: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579790/variables/variables
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:107: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
widget_view.render_fairness_indicator(eval_result=eval_result_random_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

NNLM

eval_result_nnlm = embedding_fairness_result('https://tfhub.dev/google/nnlm-en-dim128/1')
Training classifier for https://tfhub.dev/google/nnlm-en-dim128/1
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182524', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:25:24.785154: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 58.637047, step = 0
INFO:tensorflow:loss = 58.637047, step = 0
INFO:tensorflow:global_step/sec: 75.6907
INFO:tensorflow:global_step/sec: 75.6907
INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec)
INFO:tensorflow:loss = 56.208035, step = 100 (1.323 sec)
INFO:tensorflow:global_step/sec: 85.4193
INFO:tensorflow:global_step/sec: 85.4193
INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec)
INFO:tensorflow:loss = 47.563675, step = 200 (1.170 sec)
INFO:tensorflow:global_step/sec: 85.3916
INFO:tensorflow:global_step/sec: 85.3916
INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec)
INFO:tensorflow:loss = 56.227097, step = 300 (1.171 sec)
INFO:tensorflow:global_step/sec: 85.7359
INFO:tensorflow:global_step/sec: 85.7359
INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec)
INFO:tensorflow:loss = 55.668434, step = 400 (1.166 sec)
INFO:tensorflow:global_step/sec: 85.6231
INFO:tensorflow:global_step/sec: 85.6231
INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec)
INFO:tensorflow:loss = 41.7245, step = 500 (1.168 sec)
INFO:tensorflow:global_step/sec: 85.1399
INFO:tensorflow:global_step/sec: 85.1399
INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec)
INFO:tensorflow:loss = 45.596313, step = 600 (1.174 sec)
INFO:tensorflow:global_step/sec: 83.6346
INFO:tensorflow:global_step/sec: 83.6346
INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec)
INFO:tensorflow:loss = 51.108143, step = 700 (1.196 sec)
INFO:tensorflow:global_step/sec: 85.4834
INFO:tensorflow:global_step/sec: 85.4834
INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec)
INFO:tensorflow:loss = 47.63583, step = 800 (1.170 sec)
INFO:tensorflow:global_step/sec: 86.7353
INFO:tensorflow:global_step/sec: 86.7353
INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec)
INFO:tensorflow:loss = 48.044117, step = 900 (1.153 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182524/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 50.57175.
INFO:tensorflow:Loss for final step: 50.57175.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2022-01-07 18:25:40.091474: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182524/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets
INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1641579940/assets
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641579940/saved_model.pb
WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641579940/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_nnlm)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'label/mean'…

Codificador de frase universal

eval_result_use = embedding_fairness_result('https://tfhub.dev/google/universal-sentence-encoder/2')
Training classifier for https://tfhub.dev/google/universal-sentence-encoder/2
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20220107-182759', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
2022-01-07 18:28:15.955057: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.228935, step = 0
INFO:tensorflow:loss = 59.228935, step = 0
INFO:tensorflow:global_step/sec: 8.64079
INFO:tensorflow:global_step/sec: 8.64079
INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec)
INFO:tensorflow:loss = 50.28162, step = 100 (11.575 sec)
INFO:tensorflow:global_step/sec: 8.72597
INFO:tensorflow:global_step/sec: 8.72597
INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec)
INFO:tensorflow:loss = 46.290745, step = 200 (11.460 sec)
INFO:tensorflow:global_step/sec: 9.02825
INFO:tensorflow:global_step/sec: 9.02825
INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec)
INFO:tensorflow:loss = 48.490734, step = 300 (11.076 sec)
INFO:tensorflow:global_step/sec: 9.01342
INFO:tensorflow:global_step/sec: 9.01342
INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec)
INFO:tensorflow:loss = 44.54372, step = 400 (11.095 sec)
INFO:tensorflow:global_step/sec: 8.952
INFO:tensorflow:global_step/sec: 8.952
INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec)
INFO:tensorflow:loss = 35.568554, step = 500 (11.171 sec)
INFO:tensorflow:global_step/sec: 9.09908
INFO:tensorflow:global_step/sec: 9.09908
INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec)
INFO:tensorflow:loss = 42.5132, step = 600 (10.990 sec)
INFO:tensorflow:global_step/sec: 9.02127
INFO:tensorflow:global_step/sec: 9.02127
INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec)
INFO:tensorflow:loss = 40.52431, step = 700 (11.085 sec)
INFO:tensorflow:global_step/sec: 9.09376
INFO:tensorflow:global_step/sec: 9.09376
INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec)
INFO:tensorflow:loss = 37.5485, step = 800 (10.996 sec)
INFO:tensorflow:global_step/sec: 9.11679
INFO:tensorflow:global_step/sec: 9.11679
INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec)
INFO:tensorflow:loss = 32.65558, step = 900 (10.968 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20220107-182759/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 46.92047.
INFO:tensorflow:Loss for final step: 46.92047.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
2022-01-07 18:30:32.176628: W tensorflow/core/common_runtime/graph_constructor.cc:1511] Importing a graph with a lower producer version 26 into an existing graph with producer version 987. Shape inference will have run different parts of the graph with different producer versions.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
WARNING:tensorflow:Trapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Classify: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Regress: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Predict: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Train: None
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval']
WARNING:tensorflow:Export includes no default signature!
WARNING:tensorflow:Export includes no default signature!
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000
INFO:tensorflow:Restoring parameters from /tmp/train/20220107-182759/model.ckpt-1000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb
INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1641580231/saved_model.pb
WARNING:absl:Tensorflow version (2.8.0-rc0) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables
INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1641580231/variables/variables
widget_view.render_fairness_indicator(eval_result=eval_result_use)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…

Comparando Embeddings

Você também pode usar os indicadores de imparcialidade para comparar os embeddings diretamente. Por exemplo, compare os modelos gerados a partir dos embeddings NNLM e USE.

widget_view.render_fairness_indicator(multi_eval_results={'nnlm': eval_result_nnlm, 'use': eval_result_use})
FairnessIndicatorViewer(evalName='nnlm', evalNameCompare='use', slicingMetrics=[{'sliceValue': 'Overall', 'sli…