![]() | ![]() | ![]() | ![]() | ![]() |
Panoramica
Fairness Indicators è una suite di strumenti basati su TensorFlow Model Analysis (TFMA) che consente una valutazione regolare delle metriche di equità nelle pipeline di prodotti. TFMA è una libreria per la valutazione di modelli di machine learning sia TensorFlow che non TensorFlow. Ti consente di valutare i tuoi modelli su grandi quantità di dati in modo distribuito, calcolare in-graph e altre metriche su diverse sezioni di dati e visualizzarli nei notebook.
Fairness Indicators è fornito con TensorFlow Data Validation (TFDV) e lo strumento What-If . L'utilizzo degli indicatori di equità ti consente di:
- Valuta le prestazioni del modello, suddivise in gruppi definiti di utenti
- Acquisisci fiducia sui risultati con intervalli di confidenza e valutazioni a più soglie
- Valuta la distribuzione dei set di dati
- Immergiti in profondità nelle singole sezioni per esplorare le cause profonde e le opportunità di miglioramento
In questo blocco appunti, utilizzerai gli indicatori di equità per risolvere i problemi di equità in un modello addestrato utilizzando il set di dati Commenti di Civil . Guarda questo video per maggiori dettagli e contesto sullo scenario del mondo reale su cui si basa, che è anche una delle principali motivazioni per la creazione di Indicatori di equità.
Set di dati
In questo taccuino lavorerai con il set di dati Civil Comments , circa 2 milioni di commenti pubblici resi pubblici dalla piattaforma Civil Comments nel 2017 per la ricerca in corso. Questo sforzo è stato sponsorizzato da Jigsaw , che ha ospitato competizioni su Kaggle per aiutare a classificare i commenti tossici e ridurre al minimo i pregiudizi del modello non intenzionali.
Ogni singolo commento di testo nel set di dati ha un'etichetta di tossicità, con l'etichetta 1 se il commento è tossico e 0 se il commento non è tossico. All'interno dei dati, un sottoinsieme di commenti è etichettato con una varietà di attributi di identità, comprese le categorie per genere, orientamento sessuale, religione e razza o etnia.
Impostare
Installa fairness-indicators
e witwidget
.
pip install -q -U pip==20.2
pip install -q fairness-indicators
pip install -q witwidget
È necessario riavviare il runtime Colab dopo l'installazione. Seleziona Runtime> Riavvia runtime dal menu Colab.
Non procedere con il resto di questo tutorial senza prima riavviare il runtime.
Importa tutte le altre librerie richieste.
import os
import tempfile
import apache_beam as beam
import numpy as np
import pandas as pd
from datetime import datetime
import pprint
import tensorflow_hub as hub
import tensorflow as tf
import tensorflow_model_analysis as tfma
import tensorflow_data_validation as tfdv
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from tensorflow_model_analysis.addons.fairness.view import widget_view
from fairness_indicators.tutorial_utils import util
from witwidget.notebook.visualization import WitConfigBuilder
from witwidget.notebook.visualization import WitWidget
Scarica e analizza i dati
Per impostazione predefinita, questo notebook scarica una versione preelaborata di questo set di dati, ma è possibile utilizzare il set di dati originale e rieseguire le fasi di elaborazione se lo si desidera. Nel set di dati originale, ogni commento è etichettato con la percentuale di valutatori che credevano che un commento corrispondesse a una particolare identità. Ad esempio, un commento potrebbe essere etichettato con quanto segue: {male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0} La fase di elaborazione raggruppa l'identità per categoria (sesso, orientamento_sessuale, ecc.) E rimuove identità con un punteggio inferiore a 0,5. Quindi l'esempio sopra sarebbe convertito nel seguente: di valutatori che credevano che un commento corrispondesse a una particolare identità. Ad esempio, il commento sarebbe etichettato con quanto segue: {gender: [female], sex_orientation: [heterosexual, homosexual_gay_or_lesbian]}
download_original_data = False
if download_original_data:
train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')
# The identity terms list will be grouped together by their categories
# (see 'IDENTITY_COLUMNS') on threshould 0.5. Only the identity term column,
# text column and label column will be kept after processing.
train_tf_file = util.convert_comments_data(train_tf_file)
validate_tf_file = util.convert_comments_data(validate_tf_file)
else:
train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord 488161280/488153424 [==============================] - 6s 0us/step Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord 324943872/324941336 [==============================] - 5s 0us/step
Utilizzare TFDV per analizzare i dati e trovare potenziali problemi in essi, come valori mancanti e squilibri dei dati, che possono portare a disparità di equità.
stats = tfdv.generate_statistics_from_tfrecord(data_location=train_tf_file)
tfdv.visualize_statistics(stats)
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
TFDV mostra che ci sono alcuni squilibri significativi nei dati che potrebbero portare a risultati di modello distorti.
L'etichetta di tossicità (il valore previsto dal modello) è sbilanciata. Solo l'8% degli esempi nel set di addestramento sono tossici, il che significa che un classificatore potrebbe ottenere un'accuratezza del 92% prevedendo che tutti i commenti sono non tossici.
Negli ambiti relativi ai termini di identità, solo 6,6k su 1,08 milioni (0,61%) di esempi di formazione riguardano l'omosessualità, e quelli relativi alla bisessualità sono ancora più rari. Ciò indica che le prestazioni su queste sezioni potrebbero risentirne a causa della mancanza di dati di addestramento.
Prepara i dati
Definire una mappa delle caratteristiche per analizzare i dati. Ogni esempio avrà un'etichetta, un testo di commento e l'identità include sexual orientation
, il gender
, la religion
, la race
e la disability
associati al testo.
BASE_DIR = tempfile.gettempdir()
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'
FEATURE_MAP = {
# Label:
LABEL: tf.io.FixedLenFeature([], tf.float32),
# Text:
TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),
# Identities:
'sexual_orientation':tf.io.VarLenFeature(tf.string),
'gender':tf.io.VarLenFeature(tf.string),
'religion':tf.io.VarLenFeature(tf.string),
'race':tf.io.VarLenFeature(tf.string),
'disability':tf.io.VarLenFeature(tf.string),
}
Successivamente, imposta una funzione di input per inserire i dati nel modello. Aggiungi una colonna di peso a ogni esempio e aumenta il peso degli esempi tossici per tenere conto dello squilibrio di classe identificato dal TFDV. Utilizzare solo le caratteristiche dell'identità durante la fase di valutazione, poiché solo i commenti vengono inseriti nel modello durante l'addestramento.
def train_input_fn():
def parse_function(serialized):
parsed_example = tf.io.parse_single_example(
serialized=serialized, features=FEATURE_MAP)
# Adds a weight column to deal with unbalanced classes.
parsed_example['weight'] = tf.add(parsed_example[LABEL], 0.1)
return (parsed_example,
parsed_example[LABEL])
train_dataset = tf.data.TFRecordDataset(
filenames=[train_tf_file]).map(parse_function).batch(512)
return train_dataset
Addestra il modello
Crea e addestra un modello di deep learning sui dati.
model_dir = os.path.join(BASE_DIR, 'train', datetime.now().strftime(
"%Y%m%d-%H%M%S"))
embedded_text_feature_column = hub.text_embedding_column(
key=TEXT_FEATURE,
module_spec='https://tfhub.dev/google/nnlm-en-dim128/1')
classifier = tf.estimator.DNNClassifier(
hidden_units=[500, 100],
weight_column='weight',
feature_columns=[embedded_text_feature_column],
optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.003),
loss_reduction=tf.losses.Reduction.SUM,
n_classes=2,
model_dir=model_dir)
classifier.train(input_fn=train_input_fn, steps=1000)
INFO:tensorflow:Using default config. INFO:tensorflow:Using default config. INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210130-101939', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210130-101939', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/head/base_head.py:517: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/head/base_head.py:517: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version. Instructions for updating: The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/adagrad.py:83: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/adagrad.py:83: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0... INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210130-101939/model.ckpt. INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210130-101939/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0... INFO:tensorflow:loss = 59.10357, step = 0 INFO:tensorflow:loss = 59.10357, step = 0 INFO:tensorflow:global_step/sec: 21.4277 INFO:tensorflow:global_step/sec: 21.4277 INFO:tensorflow:loss = 55.88044, step = 100 (4.669 sec) INFO:tensorflow:loss = 55.88044, step = 100 (4.669 sec) INFO:tensorflow:global_step/sec: 21.9772 INFO:tensorflow:global_step/sec: 21.9772 INFO:tensorflow:loss = 47.06628, step = 200 (4.550 sec) INFO:tensorflow:loss = 47.06628, step = 200 (4.550 sec) INFO:tensorflow:global_step/sec: 21.5857 INFO:tensorflow:global_step/sec: 21.5857 INFO:tensorflow:loss = 55.466003, step = 300 (4.633 sec) INFO:tensorflow:loss = 55.466003, step = 300 (4.633 sec) INFO:tensorflow:global_step/sec: 20.7493 INFO:tensorflow:global_step/sec: 20.7493 INFO:tensorflow:loss = 56.189507, step = 400 (4.820 sec) INFO:tensorflow:loss = 56.189507, step = 400 (4.820 sec) INFO:tensorflow:global_step/sec: 21.7353 INFO:tensorflow:global_step/sec: 21.7353 INFO:tensorflow:loss = 41.800144, step = 500 (4.600 sec) INFO:tensorflow:loss = 41.800144, step = 500 (4.600 sec) INFO:tensorflow:global_step/sec: 22.551 INFO:tensorflow:global_step/sec: 22.551 INFO:tensorflow:loss = 45.347168, step = 600 (4.434 sec) INFO:tensorflow:loss = 45.347168, step = 600 (4.434 sec) INFO:tensorflow:global_step/sec: 22.292 INFO:tensorflow:global_step/sec: 22.292 INFO:tensorflow:loss = 51.336346, step = 700 (4.486 sec) INFO:tensorflow:loss = 51.336346, step = 700 (4.486 sec) INFO:tensorflow:global_step/sec: 21.9374 INFO:tensorflow:global_step/sec: 21.9374 INFO:tensorflow:loss = 47.501987, step = 800 (4.559 sec) INFO:tensorflow:loss = 47.501987, step = 800 (4.559 sec) INFO:tensorflow:global_step/sec: 22.0593 INFO:tensorflow:global_step/sec: 22.0593 INFO:tensorflow:loss = 48.003296, step = 900 (4.533 sec) INFO:tensorflow:loss = 48.003296, step = 900 (4.533 sec) INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000... INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210130-101939/model.ckpt. INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210130-101939/model.ckpt. INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000... INFO:tensorflow:Loss for final step: 50.98375. INFO:tensorflow:Loss for final step: 50.98375. <tensorflow_estimator.python.estimator.canned.dnn.DNNClassifierV2 at 0x7f76540f3a20>
Analizza il modello
Dopo aver ottenuto il modello addestrato, analizzalo per calcolare le metriche di equità utilizzando TFMA e indicatori di equità. Inizia esportando il modello come SavedModel .
Esporta SavedModel
def eval_input_receiver_fn():
serialized_tf_example = tf.compat.v1.placeholder(
dtype=tf.string, shape=[None], name='input_example_placeholder')
# This *must* be a dictionary containing a single key 'examples', which
# points to the input placeholder.
receiver_tensors = {'examples': serialized_tf_example}
features = tf.io.parse_example(serialized_tf_example, FEATURE_MAP)
features['weight'] = tf.ones_like(features[LABEL])
return tfma.export.EvalInputReceiver(
features=features,
receiver_tensors=receiver_tensors,
labels=features[LABEL])
tfma_export_dir = tfma.export.export_eval_savedmodel(
estimator=classifier,
export_dir_base=os.path.join(BASE_DIR, 'tfma_eval_model'),
eval_input_receiver_fn=eval_input_receiver_fn)
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20210130-101939/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20210130-101939/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1612002036/assets INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1612002036/assets INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1612002036/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1612002036/saved_model.pb
Metriche di equità di calcolo
Seleziona l'identità per cui calcolare le metriche e se eseguire con intervalli di confidenza utilizzando il menu a discesa nel pannello a destra.
Opzioni di calcolo degli indicatori di equità
tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')
slice_selection = 'sexual_orientation'
print(f'Slice selection: {slice_selection}')
compute_confidence_intervals = False
print(f'Compute confidence intervals: {compute_confidence_intervals}')
# Define slices that you want the evaluation to run on.
slice_spec = [
tfma.slicer.SingleSliceSpec(), # Overall slice
tfma.slicer.SingleSliceSpec(columns=[slice_selection]),
]
# Add the fairness metrics.
add_metrics_callbacks = [
tfma.post_export_metrics.fairness_indicators(
thresholds=[0.1, 0.3, 0.5, 0.7, 0.9],
labels_key=LABEL
)
]
eval_shared_model = tfma.default_eval_shared_model(
eval_saved_model_path=tfma_export_dir,
add_metrics_callbacks=add_metrics_callbacks)
# Run the fairness evaluation.
with beam.Pipeline() as pipeline:
_ = (
pipeline
| 'ReadData' >> beam.io.ReadFromTFRecord(validate_tf_file)
| 'ExtractEvaluateAndWriteResults' >>
tfma.ExtractEvaluateAndWriteResults(
eval_shared_model=eval_shared_model,
slice_spec=slice_spec,
compute_confidence_intervals=compute_confidence_intervals,
output_path=tfma_eval_result_path)
)
eval_result = tfma.load_eval_result(output_path=tfma_eval_result_path)
Slice selection: sexual_orientation Compute confidence intervals: False WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1612002036/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1612002036/variables/variables WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching:
Visualizza i dati utilizzando lo strumento What-if
In questa sezione, utilizzerai l'interfaccia visiva interattiva dello strumento What-If per esplorare e manipolare i dati a livello microscopico.
Ogni punto del grafico a dispersione nel pannello di destra rappresenta uno degli esempi nel sottoinsieme caricato nello strumento. Fare clic su uno dei punti per visualizzare i dettagli su questo particolare esempio nel pannello di sinistra. Vengono mostrati il testo del commento, la tossicità della verità fondamentale e le identità applicabili. Nella parte inferiore di questo pannello di sinistra, puoi vedere i risultati dell'inferenza dal modello che hai appena addestrato.
Modificare il testo dell'esempio e quindi fare clic sul pulsante Esegui inferenza per visualizzare in che modo le modifiche hanno causato il cambiamento della previsione di tossicità percepita.
DEFAULT_MAX_EXAMPLES = 1000
# Load 100000 examples in memory. When first rendered,
# What-If Tool should only display 1000 of these due to browser constraints.
def wit_dataset(file, num_examples=100000):
dataset = tf.data.TFRecordDataset(
filenames=[file]).take(num_examples)
return [tf.train.Example.FromString(d.numpy()) for d in dataset]
wit_data = wit_dataset(train_tf_file)
config_builder = WitConfigBuilder(wit_data[:DEFAULT_MAX_EXAMPLES]).set_estimator_and_feature_spec(
classifier, FEATURE_MAP).set_label_vocab(['non-toxicity', LABEL]).set_target_feature(LABEL)
wit = WitWidget(config_builder)
Indicatori di equità di rendering
Eseguire il rendering del widget Indicatori di equità con i risultati della valutazione esportati.
Di seguito vedrai grafici a barre che mostrano le prestazioni di ciascuna porzione di dati sulle metriche selezionate. È possibile regolare la sezione di confronto della linea di base e le soglie visualizzate utilizzando i menu a discesa nella parte superiore della visualizzazione.
Il widget Indicatore di equità è integrato con lo strumento What-If reso sopra. Se selezioni una sezione dei dati nel grafico a barre, lo strumento What-If si aggiornerà per mostrarti esempi dalla sezione selezionata. Quando i dati vengono ricaricati nello strumento What-If sopra, prova a modificare Color By in tossicità . Questo può darti una comprensione visiva dell'equilibrio della tossicità degli esempi per fetta.
event_handlers={'slice-selected':
wit.create_selection_callback(wit_data, DEFAULT_MAX_EXAMPLES)}
widget_view.render_fairness_indicator(eval_result=eval_result,
slicing_column=slice_selection,
event_handlers=event_handlers
)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'auc_precisi…
Con questo particolare set di dati e attività, tassi di falsi positivi e falsi negativi sistematicamente più elevati per determinate identità possono portare a conseguenze negative. Ad esempio, in un sistema di moderazione dei contenuti, un tasso di falsi positivi superiore al totale per un determinato gruppo può portare a tacere quelle voci. Pertanto, è importante valutare regolarmente questi tipi di criteri durante lo sviluppo e il miglioramento dei modelli e utilizzare strumenti come gli indicatori di equità, TFDV e WIT per chiarire potenziali problemi. Dopo aver identificato i problemi di equità, puoi sperimentare nuove origini dati, bilanciamento dei dati o altre tecniche per migliorare le prestazioni sui gruppi con prestazioni insufficienti.
Vedere qui per ulteriori informazioni e indicazioni su come utilizzare gli indicatori di equità.
Usa i risultati della valutazione dell'equità
L'oggetto eval_result
, reso sopra in render_fairness_indicator()
, ha una propria API che puoi sfruttare per leggere i risultati TFMA nei tuoi programmi.
Ottieni sezioni e metriche valutate
Utilizzare get_slice_names()
e get_metric_names()
per ottenere rispettivamente le sezioni e le metriche valutate.
pp = pprint.PrettyPrinter()
print("Slices:")
pp.pprint(eval_result.get_slice_names())
print("\nMetrics:")
pp.pprint(eval_result.get_metric_names())
Slices: [(), (('sexual_orientation', 'homosexual_gay_or_lesbian'),), (('sexual_orientation', 'heterosexual'),), (('sexual_orientation', 'bisexual'),), (('sexual_orientation', 'other_sexual_orientation'),)] Metrics: ['post_export_metrics/true_positive_rate@0.90', 'post_export_metrics/false_omission_rate@0.50', 'post_export_metrics/negative_rate@0.70', 'post_export_metrics/negative_rate@0.90', 'post_export_metrics/false_positive_rate@0.70', 'post_export_metrics/false_omission_rate@0.70', 'post_export_metrics/false_discovery_rate@0.70', 'accuracy', 'prediction/mean', 'post_export_metrics/false_omission_rate@0.30', 'post_export_metrics/example_count', 'post_export_metrics/false_negative_rate@0.30', 'post_export_metrics/fairness/confusion_matrix_at_thresholds', 'average_loss', 'post_export_metrics/false_positive_rate@0.30', 'post_export_metrics/false_negative_rate@0.10', 'label/mean', 'post_export_metrics/positive_rate@0.70', 'auc', 'post_export_metrics/true_negative_rate@0.50', 'precision', 'post_export_metrics/false_positive_rate@0.90', 'post_export_metrics/positive_rate@0.10', 'post_export_metrics/false_omission_rate@0.90', 'post_export_metrics/false_discovery_rate@0.10', 'post_export_metrics/false_negative_rate@0.70', 'post_export_metrics/positive_rate@0.50', 'post_export_metrics/false_discovery_rate@0.30', 'post_export_metrics/positive_rate@0.30', 'post_export_metrics/negative_rate@0.30', 'post_export_metrics/negative_rate@0.50', 'post_export_metrics/false_discovery_rate@0.90', 'post_export_metrics/true_negative_rate@0.10', 'auc_precision_recall', 'post_export_metrics/true_positive_rate@0.50', 'accuracy_baseline', 'post_export_metrics/false_positive_rate@0.10', 'post_export_metrics/false_negative_rate@0.50', 'post_export_metrics/false_omission_rate@0.10', 'post_export_metrics/false_discovery_rate@0.50', 'post_export_metrics/false_negative_rate@0.90', 'post_export_metrics/true_positive_rate@0.70', 'post_export_metrics/true_positive_rate@0.10', 'post_export_metrics/true_negative_rate@0.30', 'post_export_metrics/true_positive_rate@0.30', 'post_export_metrics/false_positive_rate@0.50', 'post_export_metrics/true_negative_rate@0.90', 'post_export_metrics/positive_rate@0.90', 'post_export_metrics/true_negative_rate@0.70', 'post_export_metrics/negative_rate@0.10', 'recall']
Utilizzare get_metrics_for_slice()
per ottenere le metriche per una particolare sezione come dizionario che associa i nomi delle metriche ai valori delle metriche .
baseline_slice = ()
heterosexual_slice = (('sexual_orientation', 'heterosexual'),)
print("Baseline metric values:")
pp.pprint(eval_result.get_metrics_for_slice(baseline_slice))
print("\nHeterosexual metric values:")
pp.pprint(eval_result.get_metrics_for_slice(heterosexual_slice))
Baseline metric values: {'accuracy': {'doubleValue': 0.7191107273101807}, 'accuracy_baseline': {'doubleValue': 0.9198060631752014}, 'auc': {'doubleValue': 0.7972093224525452}, 'auc_precision_recall': {'doubleValue': 0.3029831349849701}, 'average_loss': {'doubleValue': 0.5623680949211121}, 'label/mean': {'doubleValue': 0.08019392192363739}, 'post_export_metrics/example_count': {'doubleValue': 721950.0}, 'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 340.0}, 'boundedFalsePositives': {'value': 612773.0}, 'boundedPrecision': {'value': 0.08586231619119644}, 'boundedRecall': {'value': 0.9941273927688599}, 'boundedTrueNegatives': {'value': 51281.0}, 'boundedTruePositives': {'value': 57556.0}, 'falseNegatives': 340.0, 'falsePositives': 612773.0, 'precision': 0.08586231619119644, 'recall': 0.9941273927688599, 'tDistributionFalseNegatives': {'unsampledValue': 340.0}, 'tDistributionFalsePositives': {'unsampledValue': 612773.0}, 'tDistributionPrecision': {'unsampledValue': 0.08586231619119644}, 'tDistributionRecall': {'unsampledValue': 0.9941273927688599}, 'tDistributionTrueNegatives': {'unsampledValue': 51281.0}, 'tDistributionTruePositives': {'unsampledValue': 57556.0}, 'threshold': 0.10000000149011612, 'trueNegatives': 51281.0, 'truePositives': 57556.0}, {'boundedFalseNegatives': {'value': 4971.0}, 'boundedFalsePositives': {'value': 385684.0}, 'boundedPrecision': {'value': 0.12066555768251419}, 'boundedRecall': {'value': 0.9141391515731812}, 'boundedTrueNegatives': {'value': 278370.0}, 'boundedTruePositives': {'value': 52925.0}, 'falseNegatives': 4971.0, 'falsePositives': 385684.0, 'precision': 0.12066555768251419, 'recall': 0.9141391515731812, 'tDistributionFalseNegatives': {'unsampledValue': 4971.0}, 'tDistributionFalsePositives': {'unsampledValue': 385684.0}, 'tDistributionPrecision': {'unsampledValue': 0.12066555768251419}, 'tDistributionRecall': {'unsampledValue': 0.9141391515731812}, 'tDistributionTrueNegatives': {'unsampledValue': 278370.0}, 'tDistributionTruePositives': {'unsampledValue': 52925.0}, 'threshold': 0.30000001192092896, 'trueNegatives': 278370.0, 'truePositives': 52925.0}, {'boundedFalseNegatives': {'value': 15684.0}, 'boundedFalsePositives': {'value': 187104.0}, 'boundedPrecision': {'value': 0.18407785892486572}, 'boundedRecall': {'value': 0.7291004657745361}, 'boundedTrueNegatives': {'value': 476950.0}, 'boundedTruePositives': {'value': 42212.0}, 'falseNegatives': 15684.0, 'falsePositives': 187104.0, 'precision': 0.18407785892486572, 'recall': 0.7291004657745361, 'tDistributionFalseNegatives': {'unsampledValue': 15684.0}, 'tDistributionFalsePositives': {'unsampledValue': 187104.0}, 'tDistributionPrecision': {'unsampledValue': 0.18407785892486572}, 'tDistributionRecall': {'unsampledValue': 0.7291004657745361}, 'tDistributionTrueNegatives': {'unsampledValue': 476950.0}, 'tDistributionTruePositives': {'unsampledValue': 42212.0}, 'threshold': 0.5, 'trueNegatives': 476950.0, 'truePositives': 42212.0}, {'boundedFalseNegatives': {'value': 31392.0}, 'boundedFalsePositives': {'value': 64497.0}, 'boundedPrecision': {'value': 0.291249543428421}, 'boundedRecall': {'value': 0.4577863812446594}, 'boundedTrueNegatives': {'value': 599557.0}, 'boundedTruePositives': {'value': 26504.0}, 'falseNegatives': 31392.0, 'falsePositives': 64497.0, 'precision': 0.291249543428421, 'recall': 0.4577863812446594, 'tDistributionFalseNegatives': {'unsampledValue': 31392.0}, 'tDistributionFalsePositives': {'unsampledValue': 64497.0}, 'tDistributionPrecision': {'unsampledValue': 0.291249543428421}, 'tDistributionRecall': {'unsampledValue': 0.4577863812446594}, 'tDistributionTrueNegatives': {'unsampledValue': 599557.0}, 'tDistributionTruePositives': {'unsampledValue': 26504.0}, 'threshold': 0.699999988079071, 'trueNegatives': 599557.0, 'truePositives': 26504.0}, {'boundedFalseNegatives': {'value': 51089.0}, 'boundedFalsePositives': {'value': 6354.0}, 'boundedPrecision': {'value': 0.5172099471092224}, 'boundedRecall': {'value': 0.11757288873195648}, 'boundedTrueNegatives': {'value': 657700.0}, 'boundedTruePositives': {'value': 6807.0}, 'falseNegatives': 51089.0, 'falsePositives': 6354.0, 'precision': 0.5172099471092224, 'recall': 0.11757288873195648, 'tDistributionFalseNegatives': {'unsampledValue': 51089.0}, 'tDistributionFalsePositives': {'unsampledValue': 6354.0}, 'tDistributionPrecision': {'unsampledValue': 0.5172099471092224}, 'tDistributionRecall': {'unsampledValue': 0.11757288873195648}, 'tDistributionTrueNegatives': {'unsampledValue': 657700.0}, 'tDistributionTruePositives': {'unsampledValue': 6807.0}, 'threshold': 0.8999999761581421, 'trueNegatives': 657700.0, 'truePositives': 6807.0}]} }, 'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.9141376614570618}, 'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.8793344497680664}, 'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.8159221410751343}, 'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.7087504267692566}, 'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4827900528907776}, 'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0058725993148982525}, 'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08586085587739944}, 'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.27089953422546387}, 'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.5422136187553406}, 'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8824270963668823}, 'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.006586466915905476}, 'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.017544230446219444}, 'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.03183702379465103}, 'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.04975362494587898}, 'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.07207927852869034}, 'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9227758646011353}, 'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.5808021426200867}, 'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.28176021575927734}, 'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.09712613373994827}, 'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.009568499401211739}, 'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.07150217890739441}, 'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.3924662470817566}, 'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.6823658347129822}, 'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.8739510774612427}, 'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9817702174186707}, 'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9284977912902832}, 'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.6075337529182434}, 'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.3176341950893402}, 'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.12604889273643494}, 'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.01822979375720024}, 'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.07722414284944534}, 'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.41919782757759094}, 'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.7182397842407227}, 'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.9028738737106323}, 'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9904314875602722}, 'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 0.9941273927688599}, 'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9141391515731812}, 'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7291004657745361}, 'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.4577863812446594}, 'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.11757288873195648}, 'precision': {'doubleValue': 0.18407785892486572}, 'prediction/mean': {'doubleValue': 0.40011298656463623}, 'recall': {'doubleValue': 0.7291004657745361} } Heterosexual metric values: {'accuracy': {'doubleValue': 0.5325203537940979}, 'accuracy_baseline': {'doubleValue': 0.7601625919342041}, 'auc': {'doubleValue': 0.6657300591468811}, 'auc_precision_recall': {'doubleValue': 0.40228524804115295}, 'average_loss': {'doubleValue': 0.8246824741363525}, 'label/mean': {'doubleValue': 0.2398373931646347}, 'post_export_metrics/example_count': {'doubleValue': 492.0}, 'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0}, 'boundedFalsePositives': {'value': 362.0}, 'boundedPrecision': {'value': 0.24583333730697632}, 'boundedRecall': {'value': 1.0}, 'boundedTrueNegatives': {'value': 12.0}, 'boundedTruePositives': {'value': 118.0}, 'falsePositives': 362.0, 'precision': 0.24583333730697632, 'recall': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 0.0}, 'tDistributionFalsePositives': {'unsampledValue': 362.0}, 'tDistributionPrecision': {'unsampledValue': 0.24583333730697632}, 'tDistributionRecall': {'unsampledValue': 1.0}, 'tDistributionTrueNegatives': {'unsampledValue': 12.0}, 'tDistributionTruePositives': {'unsampledValue': 118.0}, 'threshold': 0.10000000149011612, 'trueNegatives': 12.0, 'truePositives': 118.0}, {'boundedFalseNegatives': {'value': 10.0}, 'boundedFalsePositives': {'value': 289.0}, 'boundedPrecision': {'value': 0.27204030752182007}, 'boundedRecall': {'value': 0.9152542352676392}, 'boundedTrueNegatives': {'value': 85.0}, 'boundedTruePositives': {'value': 108.0}, 'falseNegatives': 10.0, 'falsePositives': 289.0, 'precision': 0.27204030752182007, 'recall': 0.9152542352676392, 'tDistributionFalseNegatives': {'unsampledValue': 10.0}, 'tDistributionFalsePositives': {'unsampledValue': 289.0}, 'tDistributionPrecision': {'unsampledValue': 0.27204030752182007}, 'tDistributionRecall': {'unsampledValue': 0.9152542352676392}, 'tDistributionTrueNegatives': {'unsampledValue': 85.0}, 'tDistributionTruePositives': {'unsampledValue': 108.0}, 'threshold': 0.30000001192092896, 'trueNegatives': 85.0, 'truePositives': 108.0}, {'boundedFalseNegatives': {'value': 31.0}, 'boundedFalsePositives': {'value': 199.0}, 'boundedPrecision': {'value': 0.3041957914829254}, 'boundedRecall': {'value': 0.7372881174087524}, 'boundedTrueNegatives': {'value': 175.0}, 'boundedTruePositives': {'value': 87.0}, 'falseNegatives': 31.0, 'falsePositives': 199.0, 'precision': 0.3041957914829254, 'recall': 0.7372881174087524, 'tDistributionFalseNegatives': {'unsampledValue': 31.0}, 'tDistributionFalsePositives': {'unsampledValue': 199.0}, 'tDistributionPrecision': {'unsampledValue': 0.3041957914829254}, 'tDistributionRecall': {'unsampledValue': 0.7372881174087524}, 'tDistributionTrueNegatives': {'unsampledValue': 175.0}, 'tDistributionTruePositives': {'unsampledValue': 87.0}, 'threshold': 0.5, 'trueNegatives': 175.0, 'truePositives': 87.0}, {'boundedFalseNegatives': {'value': 58.0}, 'boundedFalsePositives': {'value': 111.0}, 'boundedPrecision': {'value': 0.35087719559669495}, 'boundedRecall': {'value': 0.508474588394165}, 'boundedTrueNegatives': {'value': 263.0}, 'boundedTruePositives': {'value': 60.0}, 'falseNegatives': 58.0, 'falsePositives': 111.0, 'precision': 0.35087719559669495, 'recall': 0.508474588394165, 'tDistributionFalseNegatives': {'unsampledValue': 58.0}, 'tDistributionFalsePositives': {'unsampledValue': 111.0}, 'tDistributionPrecision': {'unsampledValue': 0.35087719559669495}, 'tDistributionRecall': {'unsampledValue': 0.508474588394165}, 'tDistributionTrueNegatives': {'unsampledValue': 263.0}, 'tDistributionTruePositives': {'unsampledValue': 60.0}, 'threshold': 0.699999988079071, 'trueNegatives': 263.0, 'truePositives': 60.0}, {'boundedFalseNegatives': {'value': 97.0}, 'boundedFalsePositives': {'value': 16.0}, 'boundedPrecision': {'value': 0.5675675868988037}, 'boundedRecall': {'value': 0.17796610295772552}, 'boundedTrueNegatives': {'value': 358.0}, 'boundedTruePositives': {'value': 21.0}, 'falseNegatives': 97.0, 'falsePositives': 16.0, 'precision': 0.5675675868988037, 'recall': 0.17796610295772552, 'tDistributionFalseNegatives': {'unsampledValue': 97.0}, 'tDistributionFalsePositives': {'unsampledValue': 16.0}, 'tDistributionPrecision': {'unsampledValue': 0.5675675868988037}, 'tDistributionRecall': {'unsampledValue': 0.17796610295772552}, 'tDistributionTrueNegatives': {'unsampledValue': 358.0}, 'tDistributionTruePositives': {'unsampledValue': 21.0}, 'threshold': 0.8999999761581421, 'trueNegatives': 358.0, 'truePositives': 21.0}]} }, 'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7541666626930237}, 'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.7279596924781799}, 'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6958041787147522}, 'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.6491228342056274}, 'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4324324429035187}, 'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08474576473236084}, 'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.26271185278892517}, 'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.49152541160583496}, 'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8220338821411133}, 'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.10526315867900848}, 'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.15048544108867645}, 'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.180685356259346}, 'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.2131868153810501}, 'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9679144620895386}, 'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7727272510528564}, 'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.5320855379104614}, 'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.2967914342880249}, 'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.04278074949979782}, 'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.024390242993831635}, 'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.19308942556381226}, 'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.4186991751194}, 'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6524389982223511}, 'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9247967600822449}, 'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9756097793579102}, 'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.8069105744361877}, 'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5813007950782776}, 'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.34756097197532654}, 'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.07520325481891632}, 'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.03208556026220322}, 'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.22727273404598236}, 'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.4679144322872162}, 'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7032085657119751}, 'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9572192430496216}, 'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9152542352676392}, 'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7372881174087524}, 'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.508474588394165}, 'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.17796610295772552}, 'precision': {'doubleValue': 0.3041957914829254}, 'prediction/mean': {'doubleValue': 0.5550199747085571}, 'recall': {'doubleValue': 0.7372881174087524} }
Utilizzare get_metrics_for_all_slices()
per ottenere le metriche per tutte le sezioni come un dizionario che associa ciascuna sezione al dizionario delle metriche corrispondente ottenuto eseguendo get_metrics_for_slice()
su di esso.
pp.pprint(eval_result.get_metrics_for_all_slices())
{(): {'accuracy': {'doubleValue': 0.7191107273101807}, 'accuracy_baseline': {'doubleValue': 0.9198060631752014}, 'auc': {'doubleValue': 0.7972093224525452}, 'auc_precision_recall': {'doubleValue': 0.3029831349849701}, 'average_loss': {'doubleValue': 0.5623680949211121}, 'label/mean': {'doubleValue': 0.08019392192363739}, 'post_export_metrics/example_count': {'doubleValue': 721950.0}, 'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 340.0}, 'boundedFalsePositives': {'value': 612773.0}, 'boundedPrecision': {'value': 0.08586231619119644}, 'boundedRecall': {'value': 0.9941273927688599}, 'boundedTrueNegatives': {'value': 51281.0}, 'boundedTruePositives': {'value': 57556.0}, 'falseNegatives': 340.0, 'falsePositives': 612773.0, 'precision': 0.08586231619119644, 'recall': 0.9941273927688599, 'tDistributionFalseNegatives': {'unsampledValue': 340.0}, 'tDistributionFalsePositives': {'unsampledValue': 612773.0}, 'tDistributionPrecision': {'unsampledValue': 0.08586231619119644}, 'tDistributionRecall': {'unsampledValue': 0.9941273927688599}, 'tDistributionTrueNegatives': {'unsampledValue': 51281.0}, 'tDistributionTruePositives': {'unsampledValue': 57556.0}, 'threshold': 0.10000000149011612, 'trueNegatives': 51281.0, 'truePositives': 57556.0}, {'boundedFalseNegatives': {'value': 4971.0}, 'boundedFalsePositives': {'value': 385684.0}, 'boundedPrecision': {'value': 0.12066555768251419}, 'boundedRecall': {'value': 0.9141391515731812}, 'boundedTrueNegatives': {'value': 278370.0}, 'boundedTruePositives': {'value': 52925.0}, 'falseNegatives': 4971.0, 'falsePositives': 385684.0, 'precision': 0.12066555768251419, 'recall': 0.9141391515731812, 'tDistributionFalseNegatives': {'unsampledValue': 4971.0}, 'tDistributionFalsePositives': {'unsampledValue': 385684.0}, 'tDistributionPrecision': {'unsampledValue': 0.12066555768251419}, 'tDistributionRecall': {'unsampledValue': 0.9141391515731812}, 'tDistributionTrueNegatives': {'unsampledValue': 278370.0}, 'tDistributionTruePositives': {'unsampledValue': 52925.0}, 'threshold': 0.30000001192092896, 'trueNegatives': 278370.0, 'truePositives': 52925.0}, {'boundedFalseNegatives': {'value': 15684.0}, 'boundedFalsePositives': {'value': 187104.0}, 'boundedPrecision': {'value': 0.18407785892486572}, 'boundedRecall': {'value': 0.7291004657745361}, 'boundedTrueNegatives': {'value': 476950.0}, 'boundedTruePositives': {'value': 42212.0}, 'falseNegatives': 15684.0, 'falsePositives': 187104.0, 'precision': 0.18407785892486572, 'recall': 0.7291004657745361, 'tDistributionFalseNegatives': {'unsampledValue': 15684.0}, 'tDistributionFalsePositives': {'unsampledValue': 187104.0}, 'tDistributionPrecision': {'unsampledValue': 0.18407785892486572}, 'tDistributionRecall': {'unsampledValue': 0.7291004657745361}, 'tDistributionTrueNegatives': {'unsampledValue': 476950.0}, 'tDistributionTruePositives': {'unsampledValue': 42212.0}, 'threshold': 0.5, 'trueNegatives': 476950.0, 'truePositives': 42212.0}, {'boundedFalseNegatives': {'value': 31392.0}, 'boundedFalsePositives': {'value': 64497.0}, 'boundedPrecision': {'value': 0.291249543428421}, 'boundedRecall': {'value': 0.4577863812446594}, 'boundedTrueNegatives': {'value': 599557.0}, 'boundedTruePositives': {'value': 26504.0}, 'falseNegatives': 31392.0, 'falsePositives': 64497.0, 'precision': 0.291249543428421, 'recall': 0.4577863812446594, 'tDistributionFalseNegatives': {'unsampledValue': 31392.0}, 'tDistributionFalsePositives': {'unsampledValue': 64497.0}, 'tDistributionPrecision': {'unsampledValue': 0.291249543428421}, 'tDistributionRecall': {'unsampledValue': 0.4577863812446594}, 'tDistributionTrueNegatives': {'unsampledValue': 599557.0}, 'tDistributionTruePositives': {'unsampledValue': 26504.0}, 'threshold': 0.699999988079071, 'trueNegatives': 599557.0, 'truePositives': 26504.0}, {'boundedFalseNegatives': {'value': 51089.0}, 'boundedFalsePositives': {'value': 6354.0}, 'boundedPrecision': {'value': 0.5172099471092224}, 'boundedRecall': {'value': 0.11757288873195648}, 'boundedTrueNegatives': {'value': 657700.0}, 'boundedTruePositives': {'value': 6807.0}, 'falseNegatives': 51089.0, 'falsePositives': 6354.0, 'precision': 0.5172099471092224, 'recall': 0.11757288873195648, 'tDistributionFalseNegatives': {'unsampledValue': 51089.0}, 'tDistributionFalsePositives': {'unsampledValue': 6354.0}, 'tDistributionPrecision': {'unsampledValue': 0.5172099471092224}, 'tDistributionRecall': {'unsampledValue': 0.11757288873195648}, 'tDistributionTrueNegatives': {'unsampledValue': 657700.0}, 'tDistributionTruePositives': {'unsampledValue': 6807.0}, 'threshold': 0.8999999761581421, 'trueNegatives': 657700.0, 'truePositives': 6807.0}]} }, 'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.9141376614570618}, 'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.8793344497680664}, 'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.8159221410751343}, 'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.7087504267692566}, 'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4827900528907776}, 'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0058725993148982525}, 'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08586085587739944}, 'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.27089953422546387}, 'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.5422136187553406}, 'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8824270963668823}, 'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.006586466915905476}, 'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.017544230446219444}, 'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.03183702379465103}, 'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.04975362494587898}, 'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.07207927852869034}, 'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9227758646011353}, 'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.5808021426200867}, 'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.28176021575927734}, 'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.09712613373994827}, 'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.009568499401211739}, 'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.07150217890739441}, 'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.3924662470817566}, 'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.6823658347129822}, 'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.8739510774612427}, 'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9817702174186707}, 'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9284977912902832}, 'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.6075337529182434}, 'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.3176341950893402}, 'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.12604889273643494}, 'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.01822979375720024}, 'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.07722414284944534}, 'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.41919782757759094}, 'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.7182397842407227}, 'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.9028738737106323}, 'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9904314875602722}, 'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 0.9941273927688599}, 'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9141391515731812}, 'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7291004657745361}, 'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.4577863812446594}, 'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.11757288873195648}, 'precision': {'doubleValue': 0.18407785892486572}, 'prediction/mean': {'doubleValue': 0.40011298656463623}, 'recall': {'doubleValue': 0.7291004657745361} }, (('sexual_orientation', 'bisexual'),): {'accuracy': {'doubleValue': 0.5431034564971924}, 'accuracy_baseline': {'doubleValue': 0.8017241358757019}, 'auc': {'doubleValue': 0.6259934902191162}, 'auc_precision_recall': {'doubleValue': 0.3277454376220703}, 'average_loss': {'doubleValue': 0.7425483465194702}, 'label/mean': {'doubleValue': 0.1982758641242981}, 'post_export_metrics/example_count': {'doubleValue': 116.0}, 'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0}, 'boundedFalsePositives': {'value': 84.0}, 'boundedPrecision': {'value': 0.21495327353477478}, 'boundedRecall': {'value': 1.0}, 'boundedTrueNegatives': {'value': 9.0}, 'boundedTruePositives': {'value': 23.0}, 'falsePositives': 84.0, 'precision': 0.21495327353477478, 'recall': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 0.0}, 'tDistributionFalsePositives': {'unsampledValue': 84.0}, 'tDistributionPrecision': {'unsampledValue': 0.21495327353477478}, 'tDistributionRecall': {'unsampledValue': 1.0}, 'tDistributionTrueNegatives': {'unsampledValue': 9.0}, 'tDistributionTruePositives': {'unsampledValue': 23.0}, 'threshold': 0.10000000149011612, 'trueNegatives': 9.0, 'truePositives': 23.0}, {'boundedFalseNegatives': {'value': 4.0}, 'boundedFalsePositives': {'value': 67.0}, 'boundedPrecision': {'value': 0.22093023359775543}, 'boundedRecall': {'value': 0.8260869383811951}, 'boundedTrueNegatives': {'value': 26.0}, 'boundedTruePositives': {'value': 19.0}, 'falseNegatives': 4.0, 'falsePositives': 67.0, 'precision': 0.22093023359775543, 'recall': 0.8260869383811951, 'tDistributionFalseNegatives': {'unsampledValue': 4.0}, 'tDistributionFalsePositives': {'unsampledValue': 67.0}, 'tDistributionPrecision': {'unsampledValue': 0.22093023359775543}, 'tDistributionRecall': {'unsampledValue': 0.8260869383811951}, 'tDistributionTrueNegatives': {'unsampledValue': 26.0}, 'tDistributionTruePositives': {'unsampledValue': 19.0}, 'threshold': 0.30000001192092896, 'trueNegatives': 26.0, 'truePositives': 19.0}, {'boundedFalseNegatives': {'value': 9.0}, 'boundedFalsePositives': {'value': 44.0}, 'boundedPrecision': {'value': 0.24137930572032928}, 'boundedRecall': {'value': 0.6086956262588501}, 'boundedTrueNegatives': {'value': 49.0}, 'boundedTruePositives': {'value': 14.0}, 'falseNegatives': 9.0, 'falsePositives': 44.0, 'precision': 0.24137930572032928, 'recall': 0.6086956262588501, 'tDistributionFalseNegatives': {'unsampledValue': 9.0}, 'tDistributionFalsePositives': {'unsampledValue': 44.0}, 'tDistributionPrecision': {'unsampledValue': 0.24137930572032928}, 'tDistributionRecall': {'unsampledValue': 0.6086956262588501}, 'tDistributionTrueNegatives': {'unsampledValue': 49.0}, 'tDistributionTruePositives': {'unsampledValue': 14.0}, 'threshold': 0.5, 'trueNegatives': 49.0, 'truePositives': 14.0}, {'boundedFalseNegatives': {'value': 16.0}, 'boundedFalsePositives': {'value': 20.0}, 'boundedPrecision': {'value': 0.25925925374031067}, 'boundedRecall': {'value': 0.30434781312942505}, 'boundedTrueNegatives': {'value': 73.0}, 'boundedTruePositives': {'value': 7.0}, 'falseNegatives': 16.0, 'falsePositives': 20.0, 'precision': 0.25925925374031067, 'recall': 0.30434781312942505, 'tDistributionFalseNegatives': {'unsampledValue': 16.0}, 'tDistributionFalsePositives': {'unsampledValue': 20.0}, 'tDistributionPrecision': {'unsampledValue': 0.25925925374031067}, 'tDistributionRecall': {'unsampledValue': 0.30434781312942505}, 'tDistributionTrueNegatives': {'unsampledValue': 73.0}, 'tDistributionTruePositives': {'unsampledValue': 7.0}, 'threshold': 0.699999988079071, 'trueNegatives': 73.0, 'truePositives': 7.0}, {'boundedFalseNegatives': {'value': 22.0}, 'boundedFalsePositives': {'value': 0.0}, 'boundedPrecision': {'value': 1.0}, 'boundedRecall': {'value': 0.043478261679410934}, 'boundedTrueNegatives': {'value': 93.0}, 'boundedTruePositives': {'value': 1.0}, 'falseNegatives': 22.0, 'precision': 1.0, 'recall': 0.043478261679410934, 'tDistributionFalseNegatives': {'unsampledValue': 22.0}, 'tDistributionFalsePositives': {'unsampledValue': 0.0}, 'tDistributionPrecision': {'unsampledValue': 1.0}, 'tDistributionRecall': {'unsampledValue': 0.043478261679410934}, 'tDistributionTrueNegatives': {'unsampledValue': 93.0}, 'tDistributionTruePositives': {'unsampledValue': 1.0}, 'threshold': 0.8999999761581421, 'trueNegatives': 93.0, 'truePositives': 1.0}]} }, 'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7850467562675476}, 'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.7790697813034058}, 'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.7586206793785095}, 'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.7407407164573669}, 'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.17391304671764374}, 'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.3913043439388275}, 'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.695652186870575}, 'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.95652174949646}, 'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.13333334028720856}, 'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.1551724076271057}, 'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.17977528274059296}, 'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.19130434095859528}, 'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9032257795333862}, 'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7204301357269287}, 'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.47311827540397644}, 'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.2150537669658661}, 'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.0}, 'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.07758620381355286}, 'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.2586206793785095}, 'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.5}, 'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.767241358757019}, 'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9913793206214905}, 'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9224137663841248}, 'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.7413793206214905}, 'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5}, 'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.23275862634181976}, 'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.008620689623057842}, 'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.09677419066429138}, 'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.2795698940753937}, 'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.5268816947937012}, 'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7849462628364563}, 'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.8260869383811951}, 'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.6086956262588501}, 'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.30434781312942505}, 'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.043478261679410934}, 'precision': {'doubleValue': 0.24137930572032928}, 'prediction/mean': {'doubleValue': 0.4882037043571472}, 'recall': {'doubleValue': 0.6086956262588501} }, (('sexual_orientation', 'heterosexual'),): {'accuracy': {'doubleValue': 0.5325203537940979}, 'accuracy_baseline': {'doubleValue': 0.7601625919342041}, 'auc': {'doubleValue': 0.6657300591468811}, 'auc_precision_recall': {'doubleValue': 0.40228524804115295}, 'average_loss': {'doubleValue': 0.8246824741363525}, 'label/mean': {'doubleValue': 0.2398373931646347}, 'post_export_metrics/example_count': {'doubleValue': 492.0}, 'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0}, 'boundedFalsePositives': {'value': 362.0}, 'boundedPrecision': {'value': 0.24583333730697632}, 'boundedRecall': {'value': 1.0}, 'boundedTrueNegatives': {'value': 12.0}, 'boundedTruePositives': {'value': 118.0}, 'falsePositives': 362.0, 'precision': 0.24583333730697632, 'recall': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 0.0}, 'tDistributionFalsePositives': {'unsampledValue': 362.0}, 'tDistributionPrecision': {'unsampledValue': 0.24583333730697632}, 'tDistributionRecall': {'unsampledValue': 1.0}, 'tDistributionTrueNegatives': {'unsampledValue': 12.0}, 'tDistributionTruePositives': {'unsampledValue': 118.0}, 'threshold': 0.10000000149011612, 'trueNegatives': 12.0, 'truePositives': 118.0}, {'boundedFalseNegatives': {'value': 10.0}, 'boundedFalsePositives': {'value': 289.0}, 'boundedPrecision': {'value': 0.27204030752182007}, 'boundedRecall': {'value': 0.9152542352676392}, 'boundedTrueNegatives': {'value': 85.0}, 'boundedTruePositives': {'value': 108.0}, 'falseNegatives': 10.0, 'falsePositives': 289.0, 'precision': 0.27204030752182007, 'recall': 0.9152542352676392, 'tDistributionFalseNegatives': {'unsampledValue': 10.0}, 'tDistributionFalsePositives': {'unsampledValue': 289.0}, 'tDistributionPrecision': {'unsampledValue': 0.27204030752182007}, 'tDistributionRecall': {'unsampledValue': 0.9152542352676392}, 'tDistributionTrueNegatives': {'unsampledValue': 85.0}, 'tDistributionTruePositives': {'unsampledValue': 108.0}, 'threshold': 0.30000001192092896, 'trueNegatives': 85.0, 'truePositives': 108.0}, {'boundedFalseNegatives': {'value': 31.0}, 'boundedFalsePositives': {'value': 199.0}, 'boundedPrecision': {'value': 0.3041957914829254}, 'boundedRecall': {'value': 0.7372881174087524}, 'boundedTrueNegatives': {'value': 175.0}, 'boundedTruePositives': {'value': 87.0}, 'falseNegatives': 31.0, 'falsePositives': 199.0, 'precision': 0.3041957914829254, 'recall': 0.7372881174087524, 'tDistributionFalseNegatives': {'unsampledValue': 31.0}, 'tDistributionFalsePositives': {'unsampledValue': 199.0}, 'tDistributionPrecision': {'unsampledValue': 0.3041957914829254}, 'tDistributionRecall': {'unsampledValue': 0.7372881174087524}, 'tDistributionTrueNegatives': {'unsampledValue': 175.0}, 'tDistributionTruePositives': {'unsampledValue': 87.0}, 'threshold': 0.5, 'trueNegatives': 175.0, 'truePositives': 87.0}, {'boundedFalseNegatives': {'value': 58.0}, 'boundedFalsePositives': {'value': 111.0}, 'boundedPrecision': {'value': 0.35087719559669495}, 'boundedRecall': {'value': 0.508474588394165}, 'boundedTrueNegatives': {'value': 263.0}, 'boundedTruePositives': {'value': 60.0}, 'falseNegatives': 58.0, 'falsePositives': 111.0, 'precision': 0.35087719559669495, 'recall': 0.508474588394165, 'tDistributionFalseNegatives': {'unsampledValue': 58.0}, 'tDistributionFalsePositives': {'unsampledValue': 111.0}, 'tDistributionPrecision': {'unsampledValue': 0.35087719559669495}, 'tDistributionRecall': {'unsampledValue': 0.508474588394165}, 'tDistributionTrueNegatives': {'unsampledValue': 263.0}, 'tDistributionTruePositives': {'unsampledValue': 60.0}, 'threshold': 0.699999988079071, 'trueNegatives': 263.0, 'truePositives': 60.0}, {'boundedFalseNegatives': {'value': 97.0}, 'boundedFalsePositives': {'value': 16.0}, 'boundedPrecision': {'value': 0.5675675868988037}, 'boundedRecall': {'value': 0.17796610295772552}, 'boundedTrueNegatives': {'value': 358.0}, 'boundedTruePositives': {'value': 21.0}, 'falseNegatives': 97.0, 'falsePositives': 16.0, 'precision': 0.5675675868988037, 'recall': 0.17796610295772552, 'tDistributionFalseNegatives': {'unsampledValue': 97.0}, 'tDistributionFalsePositives': {'unsampledValue': 16.0}, 'tDistributionPrecision': {'unsampledValue': 0.5675675868988037}, 'tDistributionRecall': {'unsampledValue': 0.17796610295772552}, 'tDistributionTrueNegatives': {'unsampledValue': 358.0}, 'tDistributionTruePositives': {'unsampledValue': 21.0}, 'threshold': 0.8999999761581421, 'trueNegatives': 358.0, 'truePositives': 21.0}]} }, 'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7541666626930237}, 'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.7279596924781799}, 'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6958041787147522}, 'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.6491228342056274}, 'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4324324429035187}, 'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08474576473236084}, 'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.26271185278892517}, 'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.49152541160583496}, 'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8220338821411133}, 'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.10526315867900848}, 'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.15048544108867645}, 'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.180685356259346}, 'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.2131868153810501}, 'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9679144620895386}, 'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7727272510528564}, 'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.5320855379104614}, 'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.2967914342880249}, 'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.04278074949979782}, 'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.024390242993831635}, 'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.19308942556381226}, 'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.4186991751194}, 'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6524389982223511}, 'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9247967600822449}, 'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9756097793579102}, 'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.8069105744361877}, 'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5813007950782776}, 'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.34756097197532654}, 'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.07520325481891632}, 'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.03208556026220322}, 'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.22727273404598236}, 'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.4679144322872162}, 'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7032085657119751}, 'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9572192430496216}, 'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9152542352676392}, 'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7372881174087524}, 'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.508474588394165}, 'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.17796610295772552}, 'precision': {'doubleValue': 0.3041957914829254}, 'prediction/mean': {'doubleValue': 0.5550199747085571}, 'recall': {'doubleValue': 0.7372881174087524} }, (('sexual_orientation', 'homosexual_gay_or_lesbian'),): {'accuracy': {'doubleValue': 0.5879271030426025}, 'accuracy_baseline': {'doubleValue': 0.7182232141494751}, 'auc': {'doubleValue': 0.7076172232627869}, 'auc_precision_recall': {'doubleValue': 0.4747510254383087}, 'average_loss': {'doubleValue': 0.73641037940979}, 'label/mean': {'doubleValue': 0.2817767560482025}, 'post_export_metrics/example_count': {'doubleValue': 4390.0}, 'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 2.0}, 'boundedFalsePositives': {'value': 3031.0}, 'boundedPrecision': {'value': 0.28949835896492004}, 'boundedRecall': {'value': 0.9983831644058228}, 'boundedTrueNegatives': {'value': 122.0}, 'boundedTruePositives': {'value': 1235.0}, 'falseNegatives': 2.0, 'falsePositives': 3031.0, 'precision': 0.28949835896492004, 'recall': 0.9983831644058228, 'tDistributionFalseNegatives': {'unsampledValue': 2.0}, 'tDistributionFalsePositives': {'unsampledValue': 3031.0}, 'tDistributionPrecision': {'unsampledValue': 0.28949835896492004}, 'tDistributionRecall': {'unsampledValue': 0.9983831644058228}, 'tDistributionTrueNegatives': {'unsampledValue': 122.0}, 'tDistributionTruePositives': {'unsampledValue': 1235.0}, 'threshold': 0.10000000149011612, 'trueNegatives': 122.0, 'truePositives': 1235.0}, {'boundedFalseNegatives': {'value': 76.0}, 'boundedFalsePositives': {'value': 2368.0}, 'boundedPrecision': {'value': 0.32898837327957153}, 'boundedRecall': {'value': 0.9385610222816467}, 'boundedTrueNegatives': {'value': 785.0}, 'boundedTruePositives': {'value': 1161.0}, 'falseNegatives': 76.0, 'falsePositives': 2368.0, 'precision': 0.32898837327957153, 'recall': 0.9385610222816467, 'tDistributionFalseNegatives': {'unsampledValue': 76.0}, 'tDistributionFalsePositives': {'unsampledValue': 2368.0}, 'tDistributionPrecision': {'unsampledValue': 0.32898837327957153}, 'tDistributionRecall': {'unsampledValue': 0.9385610222816467}, 'tDistributionTrueNegatives': {'unsampledValue': 785.0}, 'tDistributionTruePositives': {'unsampledValue': 1161.0}, 'threshold': 0.30000001192092896, 'trueNegatives': 785.0, 'truePositives': 1161.0}, {'boundedFalseNegatives': {'value': 281.0}, 'boundedFalsePositives': {'value': 1528.0}, 'boundedPrecision': {'value': 0.38486313819885254}, 'boundedRecall': {'value': 0.7728375196456909}, 'boundedTrueNegatives': {'value': 1625.0}, 'boundedTruePositives': {'value': 956.0}, 'falseNegatives': 281.0, 'falsePositives': 1528.0, 'precision': 0.38486313819885254, 'recall': 0.7728375196456909, 'tDistributionFalseNegatives': {'unsampledValue': 281.0}, 'tDistributionFalsePositives': {'unsampledValue': 1528.0}, 'tDistributionPrecision': {'unsampledValue': 0.38486313819885254}, 'tDistributionRecall': {'unsampledValue': 0.7728375196456909}, 'tDistributionTrueNegatives': {'unsampledValue': 1625.0}, 'tDistributionTruePositives': {'unsampledValue': 956.0}, 'threshold': 0.5, 'trueNegatives': 1625.0, 'truePositives': 956.0}, {'boundedFalseNegatives': {'value': 611.0}, 'boundedFalsePositives': {'value': 731.0}, 'boundedPrecision': {'value': 0.4613117277622223}, 'boundedRecall': {'value': 0.5060630440711975}, 'boundedTrueNegatives': {'value': 2422.0}, 'boundedTruePositives': {'value': 626.0}, 'falseNegatives': 611.0, 'falsePositives': 731.0, 'precision': 0.4613117277622223, 'recall': 0.5060630440711975, 'tDistributionFalseNegatives': {'unsampledValue': 611.0}, 'tDistributionFalsePositives': {'unsampledValue': 731.0}, 'tDistributionPrecision': {'unsampledValue': 0.4613117277622223}, 'tDistributionRecall': {'unsampledValue': 0.5060630440711975}, 'tDistributionTrueNegatives': {'unsampledValue': 2422.0}, 'tDistributionTruePositives': {'unsampledValue': 626.0}, 'threshold': 0.699999988079071, 'trueNegatives': 2422.0, 'truePositives': 626.0}, {'boundedFalseNegatives': {'value': 1072.0}, 'boundedFalsePositives': {'value': 107.0}, 'boundedPrecision': {'value': 0.6066176295280457}, 'boundedRecall': {'value': 0.1333872228860855}, 'boundedTrueNegatives': {'value': 3046.0}, 'boundedTruePositives': {'value': 165.0}, 'falseNegatives': 1072.0, 'falsePositives': 107.0, 'precision': 0.6066176295280457, 'recall': 0.1333872228860855, 'tDistributionFalseNegatives': {'unsampledValue': 1072.0}, 'tDistributionFalsePositives': {'unsampledValue': 107.0}, 'tDistributionPrecision': {'unsampledValue': 0.6066176295280457}, 'tDistributionRecall': {'unsampledValue': 0.1333872228860855}, 'tDistributionTrueNegatives': {'unsampledValue': 3046.0}, 'tDistributionTruePositives': {'unsampledValue': 165.0}, 'threshold': 0.8999999761581421, 'trueNegatives': 3046.0, 'truePositives': 165.0}]} }, 'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7105016112327576}, 'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.6710116267204285}, 'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6151368618011475}, 'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.5386883020401001}, 'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.39338234066963196}, 'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.001616814872249961}, 'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.061438966542482376}, 'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.22716249525547028}, 'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.4939369559288025}, 'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8666127920150757}, 'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.016129031777381897}, 'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.08826945722103119}, 'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.14742916822433472}, 'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.20145070552825928}, 'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.2603205442428589}, 'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9613066911697388}, 'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7510307431221008}, 'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.48461782932281494}, 'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.23184269666671753}, 'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.03393593430519104}, 'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.028246013447642326}, 'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.19612756371498108}, 'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.43416857719421387}, 'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6908884048461914}, 'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9380410313606262}, 'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9717540144920349}, 'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.8038724660873413}, 'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5658314228057861}, 'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.309111624956131}, 'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.06195899844169617}, 'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.03869330883026123}, 'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.24896924197673798}, 'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.5153821706771851}, 'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7681573033332825}, 'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9660640954971313}, 'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 0.9983831644058228}, 'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9385610222816467}, 'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7728375196456909}, 'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.5060630440711975}, 'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.1333872228860855}, 'precision': {'doubleValue': 0.38486313819885254}, 'prediction/mean': {'doubleValue': 0.5426298975944519}, 'recall': {'doubleValue': 0.7728375196456909} }, (('sexual_orientation', 'other_sexual_orientation'),): {'accuracy': {'doubleValue': 0.6000000238418579}, 'accuracy_baseline': {'doubleValue': 0.800000011920929}, 'auc': {'doubleValue': 1.0}, 'auc_precision_recall': {'doubleValue': 1.0}, 'average_loss': {'doubleValue': 0.7428082823753357}, 'label/mean': {'doubleValue': 0.20000000298023224}, 'post_export_metrics/example_count': {'doubleValue': 5.0}, 'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0}, 'boundedFalsePositives': {'value': 4.0}, 'boundedPrecision': {'value': 0.20000000298023224}, 'boundedRecall': {'value': 1.0}, 'boundedTrueNegatives': {'value': 0.0}, 'boundedTruePositives': {'value': 1.0}, 'falsePositives': 4.0, 'precision': 0.20000000298023224, 'recall': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 0.0}, 'tDistributionFalsePositives': {'unsampledValue': 4.0}, 'tDistributionPrecision': {'unsampledValue': 0.20000000298023224}, 'tDistributionRecall': {'unsampledValue': 1.0}, 'tDistributionTrueNegatives': {'unsampledValue': 0.0}, 'tDistributionTruePositives': {'unsampledValue': 1.0}, 'threshold': 0.10000000149011612, 'truePositives': 1.0}, {'boundedFalseNegatives': {'value': 0.0}, 'boundedFalsePositives': {'value': 3.0}, 'boundedPrecision': {'value': 0.25}, 'boundedRecall': {'value': 1.0}, 'boundedTrueNegatives': {'value': 1.0}, 'boundedTruePositives': {'value': 1.0}, 'falsePositives': 3.0, 'precision': 0.25, 'recall': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 0.0}, 'tDistributionFalsePositives': {'unsampledValue': 3.0}, 'tDistributionPrecision': {'unsampledValue': 0.25}, 'tDistributionRecall': {'unsampledValue': 1.0}, 'tDistributionTrueNegatives': {'unsampledValue': 1.0}, 'tDistributionTruePositives': {'unsampledValue': 1.0}, 'threshold': 0.30000001192092896, 'trueNegatives': 1.0, 'truePositives': 1.0}, {'boundedFalseNegatives': {'value': 0.0}, 'boundedFalsePositives': {'value': 2.0}, 'boundedPrecision': {'value': 0.3333333432674408}, 'boundedRecall': {'value': 1.0}, 'boundedTrueNegatives': {'value': 2.0}, 'boundedTruePositives': {'value': 1.0}, 'falsePositives': 2.0, 'precision': 0.3333333432674408, 'recall': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 0.0}, 'tDistributionFalsePositives': {'unsampledValue': 2.0}, 'tDistributionPrecision': {'unsampledValue': 0.3333333432674408}, 'tDistributionRecall': {'unsampledValue': 1.0}, 'tDistributionTrueNegatives': {'unsampledValue': 2.0}, 'tDistributionTruePositives': {'unsampledValue': 1.0}, 'threshold': 0.5, 'trueNegatives': 2.0, 'truePositives': 1.0}, {'boundedFalseNegatives': {'value': 0.0}, 'boundedFalsePositives': {'value': 1.0}, 'boundedPrecision': {'value': 0.5}, 'boundedRecall': {'value': 1.0}, 'boundedTrueNegatives': {'value': 3.0}, 'boundedTruePositives': {'value': 1.0}, 'falsePositives': 1.0, 'precision': 0.5, 'recall': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 0.0}, 'tDistributionFalsePositives': {'unsampledValue': 1.0}, 'tDistributionPrecision': {'unsampledValue': 0.5}, 'tDistributionRecall': {'unsampledValue': 1.0}, 'tDistributionTrueNegatives': {'unsampledValue': 3.0}, 'tDistributionTruePositives': {'unsampledValue': 1.0}, 'threshold': 0.699999988079071, 'trueNegatives': 3.0, 'truePositives': 1.0}, {'boundedFalseNegatives': {'value': 1.0}, 'boundedFalsePositives': {'value': 0.0}, 'boundedPrecision': {'value': 0.0}, 'boundedRecall': {'value': 0.0}, 'boundedTrueNegatives': {'value': 4.0}, 'boundedTruePositives': {'value': 0.0}, 'falseNegatives': 1.0, 'tDistributionFalseNegatives': {'unsampledValue': 1.0}, 'tDistributionFalsePositives': {'unsampledValue': 0.0}, 'tDistributionPrecision': {'unsampledValue': 0.0}, 'tDistributionRecall': {'unsampledValue': 0.0}, 'tDistributionTrueNegatives': {'unsampledValue': 4.0}, 'tDistributionTruePositives': {'unsampledValue': 0.0}, 'threshold': 0.8999999761581421, 'trueNegatives': 4.0}]} }, 'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.800000011920929}, 'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.75}, 'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6666666865348816}, 'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.5}, 'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.0}, 'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 1.0}, 'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.0}, 'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.0}, 'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.0}, 'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.20000000298023224}, 'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 1.0}, 'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.75}, 'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.5}, 'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.25}, 'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.0}, 'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.20000000298023224}, 'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.4000000059604645}, 'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6000000238418579}, 'post_export_metrics/negative_rate@0.90': {'doubleValue': 1.0}, 'post_export_metrics/positive_rate@0.10': {'doubleValue': 1.0}, 'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.800000011920929}, 'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.6000000238418579}, 'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.4000000059604645}, 'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.0}, 'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.0}, 'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.25}, 'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.5}, 'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.75}, 'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 1.0}, 'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.0}, 'precision': {'doubleValue': 0.3333333432674408}, 'prediction/mean': {'doubleValue': 0.5955214500427246}, 'recall': {'doubleValue': 1.0} } }