 Visualizza su TensorFlow.org |  Esegui in Google Colab |  Visualizza su GitHub |  Scarica taccuino |  Vedi modello TF Hub |
Panoramica
Fairness Indicators è una suite di strumenti basati su TensorFlow Model Analysis (TFMA) che consente una valutazione regolare delle metriche di equità nelle pipeline di prodotti. TFMA è una libreria per la valutazione di modelli di machine learning sia TensorFlow che non TensorFlow. Ti consente di valutare i tuoi modelli su grandi quantità di dati in modo distribuito, calcolare in-graph e altre metriche su diverse sezioni di dati e visualizzarli nei notebook.
Fairness Indicators è fornito con TensorFlow Data Validation (TFDV) e lo strumento What-If . L'utilizzo degli indicatori di equità ti consente di:
- Valuta le prestazioni del modello, suddivise in gruppi definiti di utenti
- Acquisisci fiducia sui risultati con intervalli di confidenza e valutazioni a più soglie
- Valuta la distribuzione dei set di dati
- Immergiti in profondità nelle singole sezioni per esplorare le cause profonde e le opportunità di miglioramento
In questo blocco appunti, utilizzerai gli indicatori di equità per risolvere i problemi di equità in un modello addestrato utilizzando il set di dati Commenti di Civil . Guarda questo video per maggiori dettagli e contesto sullo scenario del mondo reale su cui si basa, che è anche una delle principali motivazioni per la creazione di Indicatori di equità.
Set di dati
In questo taccuino lavorerai con il set di dati Civil Comments , circa 2 milioni di commenti pubblici resi pubblici dalla piattaforma Civil Comments nel 2017 per la ricerca in corso. Questo sforzo è stato sponsorizzato da Jigsaw , che ha ospitato competizioni su Kaggle per aiutare a classificare i commenti tossici e ridurre al minimo i pregiudizi del modello non intenzionali.
Ogni singolo commento di testo nel set di dati ha un'etichetta di tossicità, con l'etichetta 1 se il commento è tossico e 0 se il commento non è tossico. All'interno dei dati, un sottoinsieme di commenti è etichettato con una varietà di attributi di identità, comprese le categorie per genere, orientamento sessuale, religione e razza o etnia.
Impostare
Installa fairness-indicators e witwidget .
pip install -q -U pip==20.2pip install -q fairness-indicatorspip install -q witwidget
È necessario riavviare il runtime Colab dopo l'installazione. Seleziona Runtime> Riavvia runtime dal menu Colab.
Non procedere con il resto di questo tutorial senza prima riavviare il runtime.
Importa tutte le altre librerie richieste.
import os
import tempfile
import apache_beam as beam
import numpy as np
import pandas as pd
from datetime import datetime
import pprint
import tensorflow_hub as hub
import tensorflow as tf
import tensorflow_model_analysis as tfma
import tensorflow_data_validation as tfdv
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from tensorflow_model_analysis.addons.fairness.view import widget_view
from fairness_indicators.tutorial_utils import util
from witwidget.notebook.visualization import WitConfigBuilder
from witwidget.notebook.visualization import WitWidget
Scarica e analizza i dati
Per impostazione predefinita, questo notebook scarica una versione preelaborata di questo set di dati, ma è possibile utilizzare il set di dati originale e rieseguire le fasi di elaborazione se lo si desidera. Nel set di dati originale, ogni commento è etichettato con la percentuale di valutatori che credevano che un commento corrispondesse a una particolare identità. Ad esempio, un commento potrebbe essere etichettato con quanto segue: {male: 0.3, female: 1.0, transgender: 0.0, heterosexual: 0.8, homosexual_gay_or_lesbian: 1.0} La fase di elaborazione raggruppa l'identità per categoria (sesso, orientamento_sessuale, ecc.) E rimuove identità con un punteggio inferiore a 0,5. Quindi l'esempio sopra sarebbe convertito nel seguente: di valutatori che credevano che un commento corrispondesse a una particolare identità. Ad esempio, il commento sarebbe etichettato con quanto segue: {gender: [female], sex_orientation: [heterosexual, homosexual_gay_or_lesbian]}
download_original_data = False
if download_original_data:
train_tf_file = tf.keras.utils.get_file('train_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf.tfrecord')
# The identity terms list will be grouped together by their categories
# (see 'IDENTITY_COLUMNS') on threshould 0.5. Only the identity term column,
# text column and label column will be kept after processing.
train_tf_file = util.convert_comments_data(train_tf_file)
validate_tf_file = util.convert_comments_data(validate_tf_file)
else:
train_tf_file = tf.keras.utils.get_file('train_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord')
validate_tf_file = tf.keras.utils.get_file('validate_tf_processed.tfrecord',
'https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord')
Downloading data from https://storage.googleapis.com/civil_comments_dataset/train_tf_processed.tfrecord 488161280/488153424 [==============================] - 6s 0us/step Downloading data from https://storage.googleapis.com/civil_comments_dataset/validate_tf_processed.tfrecord 324943872/324941336 [==============================] - 5s 0us/step
Utilizzare TFDV per analizzare i dati e trovare potenziali problemi in essi, come valori mancanti e squilibri dei dati, che possono portare a disparità di equità.
stats = tfdv.generate_statistics_from_tfrecord(data_location=train_tf_file)
tfdv.visualize_statistics(stats)
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
TFDV mostra che ci sono alcuni squilibri significativi nei dati che potrebbero portare a risultati di modello distorti.
L'etichetta di tossicità (il valore previsto dal modello) è sbilanciata. Solo l'8% degli esempi nel set di addestramento sono tossici, il che significa che un classificatore potrebbe ottenere un'accuratezza del 92% prevedendo che tutti i commenti sono non tossici.
Negli ambiti relativi ai termini di identità, solo 6,6k su 1,08 milioni (0,61%) di esempi di formazione riguardano l'omosessualità, e quelli relativi alla bisessualità sono ancora più rari. Ciò indica che le prestazioni su queste sezioni potrebbero risentirne a causa della mancanza di dati di addestramento.
Prepara i dati
Definire una mappa delle caratteristiche per analizzare i dati. Ogni esempio avrà un'etichetta, un testo di commento e l'identità include sexual orientation , il gender , la religion , la race e la disability associati al testo.
BASE_DIR = tempfile.gettempdir()
TEXT_FEATURE = 'comment_text'
LABEL = 'toxicity'
FEATURE_MAP = {
# Label:
LABEL: tf.io.FixedLenFeature([], tf.float32),
# Text:
TEXT_FEATURE: tf.io.FixedLenFeature([], tf.string),
# Identities:
'sexual_orientation':tf.io.VarLenFeature(tf.string),
'gender':tf.io.VarLenFeature(tf.string),
'religion':tf.io.VarLenFeature(tf.string),
'race':tf.io.VarLenFeature(tf.string),
'disability':tf.io.VarLenFeature(tf.string),
}
Successivamente, imposta una funzione di input per inserire i dati nel modello. Aggiungi una colonna di peso a ogni esempio e aumenta il peso degli esempi tossici per tenere conto dello squilibrio di classe identificato dal TFDV. Utilizzare solo le caratteristiche dell'identità durante la fase di valutazione, poiché solo i commenti vengono inseriti nel modello durante l'addestramento.
def train_input_fn():
def parse_function(serialized):
parsed_example = tf.io.parse_single_example(
serialized=serialized, features=FEATURE_MAP)
# Adds a weight column to deal with unbalanced classes.
parsed_example['weight'] = tf.add(parsed_example[LABEL], 0.1)
return (parsed_example,
parsed_example[LABEL])
train_dataset = tf.data.TFRecordDataset(
filenames=[train_tf_file]).map(parse_function).batch(512)
return train_dataset
Addestra il modello
Crea e addestra un modello di deep learning sui dati.
model_dir = os.path.join(BASE_DIR, 'train', datetime.now().strftime(
"%Y%m%d-%H%M%S"))
embedded_text_feature_column = hub.text_embedding_column(
key=TEXT_FEATURE,
module_spec='https://tfhub.dev/google/nnlm-en-dim128/1')
classifier = tf.estimator.DNNClassifier(
hidden_units=[500, 100],
weight_column='weight',
feature_columns=[embedded_text_feature_column],
optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.003),
loss_reduction=tf.losses.Reduction.SUM,
n_classes=2,
model_dir=model_dir)
classifier.train(input_fn=train_input_fn, steps=1000)
INFO:tensorflow:Using default config.
INFO:tensorflow:Using default config.
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210130-101939', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Using config: {'_model_dir': '/tmp/train/20210130-101939', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/head/base_head.py:517: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/head/base_head.py:517: NumericColumn._get_dense_tensor (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column.py:2192: NumericColumn._transform_feature (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.
Instructions for updating:
The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/adagrad.py:83: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/adagrad.py:83: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210130-101939/model.ckpt.
INFO:tensorflow:Saving checkpoints for 0 into /tmp/train/20210130-101939/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:loss = 59.10357, step = 0
INFO:tensorflow:loss = 59.10357, step = 0
INFO:tensorflow:global_step/sec: 21.4277
INFO:tensorflow:global_step/sec: 21.4277
INFO:tensorflow:loss = 55.88044, step = 100 (4.669 sec)
INFO:tensorflow:loss = 55.88044, step = 100 (4.669 sec)
INFO:tensorflow:global_step/sec: 21.9772
INFO:tensorflow:global_step/sec: 21.9772
INFO:tensorflow:loss = 47.06628, step = 200 (4.550 sec)
INFO:tensorflow:loss = 47.06628, step = 200 (4.550 sec)
INFO:tensorflow:global_step/sec: 21.5857
INFO:tensorflow:global_step/sec: 21.5857
INFO:tensorflow:loss = 55.466003, step = 300 (4.633 sec)
INFO:tensorflow:loss = 55.466003, step = 300 (4.633 sec)
INFO:tensorflow:global_step/sec: 20.7493
INFO:tensorflow:global_step/sec: 20.7493
INFO:tensorflow:loss = 56.189507, step = 400 (4.820 sec)
INFO:tensorflow:loss = 56.189507, step = 400 (4.820 sec)
INFO:tensorflow:global_step/sec: 21.7353
INFO:tensorflow:global_step/sec: 21.7353
INFO:tensorflow:loss = 41.800144, step = 500 (4.600 sec)
INFO:tensorflow:loss = 41.800144, step = 500 (4.600 sec)
INFO:tensorflow:global_step/sec: 22.551
INFO:tensorflow:global_step/sec: 22.551
INFO:tensorflow:loss = 45.347168, step = 600 (4.434 sec)
INFO:tensorflow:loss = 45.347168, step = 600 (4.434 sec)
INFO:tensorflow:global_step/sec: 22.292
INFO:tensorflow:global_step/sec: 22.292
INFO:tensorflow:loss = 51.336346, step = 700 (4.486 sec)
INFO:tensorflow:loss = 51.336346, step = 700 (4.486 sec)
INFO:tensorflow:global_step/sec: 21.9374
INFO:tensorflow:global_step/sec: 21.9374
INFO:tensorflow:loss = 47.501987, step = 800 (4.559 sec)
INFO:tensorflow:loss = 47.501987, step = 800 (4.559 sec)
INFO:tensorflow:global_step/sec: 22.0593
INFO:tensorflow:global_step/sec: 22.0593
INFO:tensorflow:loss = 48.003296, step = 900 (4.533 sec)
INFO:tensorflow:loss = 48.003296, step = 900 (4.533 sec)
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 1000...
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210130-101939/model.ckpt.
INFO:tensorflow:Saving checkpoints for 1000 into /tmp/train/20210130-101939/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 1000...
INFO:tensorflow:Loss for final step: 50.98375.
INFO:tensorflow:Loss for final step: 50.98375.
<tensorflow_estimator.python.estimator.canned.dnn.DNNClassifierV2 at 0x7f76540f3a20>
Analizza il modello
Dopo aver ottenuto il modello addestrato, analizzalo per calcolare le metriche di equità utilizzando TFMA e indicatori di equità. Inizia esportando il modello come SavedModel .
Esporta SavedModel
def eval_input_receiver_fn():
serialized_tf_example = tf.compat.v1.placeholder(
dtype=tf.string, shape=[None], name='input_example_placeholder')
# This *must* be a dictionary containing a single key 'examples', which
# points to the input placeholder.
receiver_tensors = {'examples': serialized_tf_example}
features = tf.io.parse_example(serialized_tf_example, FEATURE_MAP)
features['weight'] = tf.ones_like(features[LABEL])
return tfma.export.EvalInputReceiver(
features=features,
receiver_tensors=receiver_tensors,
labels=features[LABEL])
tfma_export_dir = tfma.export.export_eval_savedmodel(
estimator=classifier,
export_dir_base=os.path.join(BASE_DIR, 'tfma_eval_model'),
eval_input_receiver_fn=eval_input_receiver_fn)
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/encoding.py:141: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Saver not created because there are no variables in the graph to restore INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Predict: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] INFO:tensorflow:Signatures INCLUDED in export for Eval: ['eval'] WARNING:tensorflow:Export includes no default signature! WARNING:tensorflow:Export includes no default signature! INFO:tensorflow:Restoring parameters from /tmp/train/20210130-101939/model.ckpt-1000 INFO:tensorflow:Restoring parameters from /tmp/train/20210130-101939/model.ckpt-1000 INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets added to graph. INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1612002036/assets INFO:tensorflow:Assets written to: /tmp/tfma_eval_model/temp-1612002036/assets INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1612002036/saved_model.pb INFO:tensorflow:SavedModel written to: /tmp/tfma_eval_model/temp-1612002036/saved_model.pb
Metriche di equità di calcolo
Seleziona l'identità per cui calcolare le metriche e se eseguire con intervalli di confidenza utilizzando il menu a discesa nel pannello a destra.
Opzioni di calcolo degli indicatori di equità
tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')
slice_selection = 'sexual_orientation'
print(f'Slice selection: {slice_selection}')
compute_confidence_intervals = False
print(f'Compute confidence intervals: {compute_confidence_intervals}')
# Define slices that you want the evaluation to run on.
slice_spec = [
tfma.slicer.SingleSliceSpec(), # Overall slice
tfma.slicer.SingleSliceSpec(columns=[slice_selection]),
]
# Add the fairness metrics.
add_metrics_callbacks = [
tfma.post_export_metrics.fairness_indicators(
thresholds=[0.1, 0.3, 0.5, 0.7, 0.9],
labels_key=LABEL
)
]
eval_shared_model = tfma.default_eval_shared_model(
eval_saved_model_path=tfma_export_dir,
add_metrics_callbacks=add_metrics_callbacks)
# Run the fairness evaluation.
with beam.Pipeline() as pipeline:
_ = (
pipeline
| 'ReadData' >> beam.io.ReadFromTFRecord(validate_tf_file)
| 'ExtractEvaluateAndWriteResults' >>
tfma.ExtractEvaluateAndWriteResults(
eval_shared_model=eval_shared_model,
slice_spec=slice_spec,
compute_confidence_intervals=compute_confidence_intervals,
output_path=tfma_eval_result_path)
)
eval_result = tfma.load_eval_result(output_path=tfma_eval_result_path)
Slice selection: sexual_orientation Compute confidence intervals: False WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:169: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0. INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1612002036/variables/variables INFO:tensorflow:Restoring parameters from /tmp/tfma_eval_model/1612002036/variables/variables WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:189: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.get_tensor_from_tensor_info or tf.compat.v1.saved_model.get_tensor_from_tensor_info. WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching:
Visualizza i dati utilizzando lo strumento What-if
In questa sezione, utilizzerai l'interfaccia visiva interattiva dello strumento What-If per esplorare e manipolare i dati a livello microscopico.
Ogni punto del grafico a dispersione nel pannello di destra rappresenta uno degli esempi nel sottoinsieme caricato nello strumento. Fare clic su uno dei punti per visualizzare i dettagli su questo particolare esempio nel pannello di sinistra. Vengono mostrati il testo del commento, la tossicità della verità fondamentale e le identità applicabili. Nella parte inferiore di questo pannello di sinistra, puoi vedere i risultati dell'inferenza dal modello che hai appena addestrato.
Modificare il testo dell'esempio e quindi fare clic sul pulsante Esegui inferenza per visualizzare in che modo le modifiche hanno causato il cambiamento della previsione di tossicità percepita.
DEFAULT_MAX_EXAMPLES = 1000
# Load 100000 examples in memory. When first rendered,
# What-If Tool should only display 1000 of these due to browser constraints.
def wit_dataset(file, num_examples=100000):
dataset = tf.data.TFRecordDataset(
filenames=[file]).take(num_examples)
return [tf.train.Example.FromString(d.numpy()) for d in dataset]
wit_data = wit_dataset(train_tf_file)
config_builder = WitConfigBuilder(wit_data[:DEFAULT_MAX_EXAMPLES]).set_estimator_and_feature_spec(
classifier, FEATURE_MAP).set_label_vocab(['non-toxicity', LABEL]).set_target_feature(LABEL)
wit = WitWidget(config_builder)
Indicatori di equità di rendering
Eseguire il rendering del widget Indicatori di equità con i risultati della valutazione esportati.
Di seguito vedrai grafici a barre che mostrano le prestazioni di ciascuna porzione di dati sulle metriche selezionate. È possibile regolare la sezione di confronto della linea di base e le soglie visualizzate utilizzando i menu a discesa nella parte superiore della visualizzazione.
Il widget Indicatore di equità è integrato con lo strumento What-If reso sopra. Se selezioni una sezione dei dati nel grafico a barre, lo strumento What-If si aggiornerà per mostrarti esempi dalla sezione selezionata. Quando i dati vengono ricaricati nello strumento What-If sopra, prova a modificare Color By in tossicità . Questo può darti una comprensione visiva dell'equilibrio della tossicità degli esempi per fetta.
event_handlers={'slice-selected':
wit.create_selection_callback(wit_data, DEFAULT_MAX_EXAMPLES)}
widget_view.render_fairness_indicator(eval_result=eval_result,
slicing_column=slice_selection,
event_handlers=event_handlers
)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'auc_precisi…
Con questo particolare set di dati e attività, tassi di falsi positivi e falsi negativi sistematicamente più elevati per determinate identità possono portare a conseguenze negative. Ad esempio, in un sistema di moderazione dei contenuti, un tasso di falsi positivi superiore al totale per un determinato gruppo può portare a tacere quelle voci. Pertanto, è importante valutare regolarmente questi tipi di criteri durante lo sviluppo e il miglioramento dei modelli e utilizzare strumenti come gli indicatori di equità, TFDV e WIT per chiarire potenziali problemi. Dopo aver identificato i problemi di equità, puoi sperimentare nuove origini dati, bilanciamento dei dati o altre tecniche per migliorare le prestazioni sui gruppi con prestazioni insufficienti.
Vedere qui per ulteriori informazioni e indicazioni su come utilizzare gli indicatori di equità.
Usa i risultati della valutazione dell'equità
L'oggetto eval_result , reso sopra in render_fairness_indicator() , ha una propria API che puoi sfruttare per leggere i risultati TFMA nei tuoi programmi.
Ottieni sezioni e metriche valutate
Utilizzare get_slice_names() e get_metric_names() per ottenere rispettivamente le sezioni e le metriche valutate.
pp = pprint.PrettyPrinter()
print("Slices:")
pp.pprint(eval_result.get_slice_names())
print("\nMetrics:")
pp.pprint(eval_result.get_metric_names())
Slices:
[(),
(('sexual_orientation', 'homosexual_gay_or_lesbian'),),
(('sexual_orientation', 'heterosexual'),),
(('sexual_orientation', 'bisexual'),),
(('sexual_orientation', 'other_sexual_orientation'),)]
Metrics:
['post_export_metrics/true_positive_rate@0.90',
'post_export_metrics/false_omission_rate@0.50',
'post_export_metrics/negative_rate@0.70',
'post_export_metrics/negative_rate@0.90',
'post_export_metrics/false_positive_rate@0.70',
'post_export_metrics/false_omission_rate@0.70',
'post_export_metrics/false_discovery_rate@0.70',
'accuracy',
'prediction/mean',
'post_export_metrics/false_omission_rate@0.30',
'post_export_metrics/example_count',
'post_export_metrics/false_negative_rate@0.30',
'post_export_metrics/fairness/confusion_matrix_at_thresholds',
'average_loss',
'post_export_metrics/false_positive_rate@0.30',
'post_export_metrics/false_negative_rate@0.10',
'label/mean',
'post_export_metrics/positive_rate@0.70',
'auc',
'post_export_metrics/true_negative_rate@0.50',
'precision',
'post_export_metrics/false_positive_rate@0.90',
'post_export_metrics/positive_rate@0.10',
'post_export_metrics/false_omission_rate@0.90',
'post_export_metrics/false_discovery_rate@0.10',
'post_export_metrics/false_negative_rate@0.70',
'post_export_metrics/positive_rate@0.50',
'post_export_metrics/false_discovery_rate@0.30',
'post_export_metrics/positive_rate@0.30',
'post_export_metrics/negative_rate@0.30',
'post_export_metrics/negative_rate@0.50',
'post_export_metrics/false_discovery_rate@0.90',
'post_export_metrics/true_negative_rate@0.10',
'auc_precision_recall',
'post_export_metrics/true_positive_rate@0.50',
'accuracy_baseline',
'post_export_metrics/false_positive_rate@0.10',
'post_export_metrics/false_negative_rate@0.50',
'post_export_metrics/false_omission_rate@0.10',
'post_export_metrics/false_discovery_rate@0.50',
'post_export_metrics/false_negative_rate@0.90',
'post_export_metrics/true_positive_rate@0.70',
'post_export_metrics/true_positive_rate@0.10',
'post_export_metrics/true_negative_rate@0.30',
'post_export_metrics/true_positive_rate@0.30',
'post_export_metrics/false_positive_rate@0.50',
'post_export_metrics/true_negative_rate@0.90',
'post_export_metrics/positive_rate@0.90',
'post_export_metrics/true_negative_rate@0.70',
'post_export_metrics/negative_rate@0.10',
'recall']
Utilizzare get_metrics_for_slice() per ottenere le metriche per una particolare sezione come dizionario che associa i nomi delle metriche ai valori delle metriche .
baseline_slice = ()
heterosexual_slice = (('sexual_orientation', 'heterosexual'),)
print("Baseline metric values:")
pp.pprint(eval_result.get_metrics_for_slice(baseline_slice))
print("\nHeterosexual metric values:")
pp.pprint(eval_result.get_metrics_for_slice(heterosexual_slice))
Baseline metric values:
{'accuracy': {'doubleValue': 0.7191107273101807},
'accuracy_baseline': {'doubleValue': 0.9198060631752014},
'auc': {'doubleValue': 0.7972093224525452},
'auc_precision_recall': {'doubleValue': 0.3029831349849701},
'average_loss': {'doubleValue': 0.5623680949211121},
'label/mean': {'doubleValue': 0.08019392192363739},
'post_export_metrics/example_count': {'doubleValue': 721950.0},
'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 340.0},
'boundedFalsePositives': {'value': 612773.0},
'boundedPrecision': {'value': 0.08586231619119644},
'boundedRecall': {'value': 0.9941273927688599},
'boundedTrueNegatives': {'value': 51281.0},
'boundedTruePositives': {'value': 57556.0},
'falseNegatives': 340.0,
'falsePositives': 612773.0,
'precision': 0.08586231619119644,
'recall': 0.9941273927688599,
'tDistributionFalseNegatives': {'unsampledValue': 340.0},
'tDistributionFalsePositives': {'unsampledValue': 612773.0},
'tDistributionPrecision': {'unsampledValue': 0.08586231619119644},
'tDistributionRecall': {'unsampledValue': 0.9941273927688599},
'tDistributionTrueNegatives': {'unsampledValue': 51281.0},
'tDistributionTruePositives': {'unsampledValue': 57556.0},
'threshold': 0.10000000149011612,
'trueNegatives': 51281.0,
'truePositives': 57556.0},
{'boundedFalseNegatives': {'value': 4971.0},
'boundedFalsePositives': {'value': 385684.0},
'boundedPrecision': {'value': 0.12066555768251419},
'boundedRecall': {'value': 0.9141391515731812},
'boundedTrueNegatives': {'value': 278370.0},
'boundedTruePositives': {'value': 52925.0},
'falseNegatives': 4971.0,
'falsePositives': 385684.0,
'precision': 0.12066555768251419,
'recall': 0.9141391515731812,
'tDistributionFalseNegatives': {'unsampledValue': 4971.0},
'tDistributionFalsePositives': {'unsampledValue': 385684.0},
'tDistributionPrecision': {'unsampledValue': 0.12066555768251419},
'tDistributionRecall': {'unsampledValue': 0.9141391515731812},
'tDistributionTrueNegatives': {'unsampledValue': 278370.0},
'tDistributionTruePositives': {'unsampledValue': 52925.0},
'threshold': 0.30000001192092896,
'trueNegatives': 278370.0,
'truePositives': 52925.0},
{'boundedFalseNegatives': {'value': 15684.0},
'boundedFalsePositives': {'value': 187104.0},
'boundedPrecision': {'value': 0.18407785892486572},
'boundedRecall': {'value': 0.7291004657745361},
'boundedTrueNegatives': {'value': 476950.0},
'boundedTruePositives': {'value': 42212.0},
'falseNegatives': 15684.0,
'falsePositives': 187104.0,
'precision': 0.18407785892486572,
'recall': 0.7291004657745361,
'tDistributionFalseNegatives': {'unsampledValue': 15684.0},
'tDistributionFalsePositives': {'unsampledValue': 187104.0},
'tDistributionPrecision': {'unsampledValue': 0.18407785892486572},
'tDistributionRecall': {'unsampledValue': 0.7291004657745361},
'tDistributionTrueNegatives': {'unsampledValue': 476950.0},
'tDistributionTruePositives': {'unsampledValue': 42212.0},
'threshold': 0.5,
'trueNegatives': 476950.0,
'truePositives': 42212.0},
{'boundedFalseNegatives': {'value': 31392.0},
'boundedFalsePositives': {'value': 64497.0},
'boundedPrecision': {'value': 0.291249543428421},
'boundedRecall': {'value': 0.4577863812446594},
'boundedTrueNegatives': {'value': 599557.0},
'boundedTruePositives': {'value': 26504.0},
'falseNegatives': 31392.0,
'falsePositives': 64497.0,
'precision': 0.291249543428421,
'recall': 0.4577863812446594,
'tDistributionFalseNegatives': {'unsampledValue': 31392.0},
'tDistributionFalsePositives': {'unsampledValue': 64497.0},
'tDistributionPrecision': {'unsampledValue': 0.291249543428421},
'tDistributionRecall': {'unsampledValue': 0.4577863812446594},
'tDistributionTrueNegatives': {'unsampledValue': 599557.0},
'tDistributionTruePositives': {'unsampledValue': 26504.0},
'threshold': 0.699999988079071,
'trueNegatives': 599557.0,
'truePositives': 26504.0},
{'boundedFalseNegatives': {'value': 51089.0},
'boundedFalsePositives': {'value': 6354.0},
'boundedPrecision': {'value': 0.5172099471092224},
'boundedRecall': {'value': 0.11757288873195648},
'boundedTrueNegatives': {'value': 657700.0},
'boundedTruePositives': {'value': 6807.0},
'falseNegatives': 51089.0,
'falsePositives': 6354.0,
'precision': 0.5172099471092224,
'recall': 0.11757288873195648,
'tDistributionFalseNegatives': {'unsampledValue': 51089.0},
'tDistributionFalsePositives': {'unsampledValue': 6354.0},
'tDistributionPrecision': {'unsampledValue': 0.5172099471092224},
'tDistributionRecall': {'unsampledValue': 0.11757288873195648},
'tDistributionTrueNegatives': {'unsampledValue': 657700.0},
'tDistributionTruePositives': {'unsampledValue': 6807.0},
'threshold': 0.8999999761581421,
'trueNegatives': 657700.0,
'truePositives': 6807.0}]} },
'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.9141376614570618},
'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.8793344497680664},
'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.8159221410751343},
'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.7087504267692566},
'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4827900528907776},
'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0058725993148982525},
'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08586085587739944},
'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.27089953422546387},
'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.5422136187553406},
'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8824270963668823},
'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.006586466915905476},
'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.017544230446219444},
'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.03183702379465103},
'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.04975362494587898},
'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.07207927852869034},
'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9227758646011353},
'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.5808021426200867},
'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.28176021575927734},
'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.09712613373994827},
'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.009568499401211739},
'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.07150217890739441},
'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.3924662470817566},
'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.6823658347129822},
'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.8739510774612427},
'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9817702174186707},
'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9284977912902832},
'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.6075337529182434},
'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.3176341950893402},
'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.12604889273643494},
'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.01822979375720024},
'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.07722414284944534},
'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.41919782757759094},
'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.7182397842407227},
'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.9028738737106323},
'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9904314875602722},
'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 0.9941273927688599},
'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9141391515731812},
'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7291004657745361},
'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.4577863812446594},
'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.11757288873195648},
'precision': {'doubleValue': 0.18407785892486572},
'prediction/mean': {'doubleValue': 0.40011298656463623},
'recall': {'doubleValue': 0.7291004657745361} }
Heterosexual metric values:
{'accuracy': {'doubleValue': 0.5325203537940979},
'accuracy_baseline': {'doubleValue': 0.7601625919342041},
'auc': {'doubleValue': 0.6657300591468811},
'auc_precision_recall': {'doubleValue': 0.40228524804115295},
'average_loss': {'doubleValue': 0.8246824741363525},
'label/mean': {'doubleValue': 0.2398373931646347},
'post_export_metrics/example_count': {'doubleValue': 492.0},
'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0},
'boundedFalsePositives': {'value': 362.0},
'boundedPrecision': {'value': 0.24583333730697632},
'boundedRecall': {'value': 1.0},
'boundedTrueNegatives': {'value': 12.0},
'boundedTruePositives': {'value': 118.0},
'falsePositives': 362.0,
'precision': 0.24583333730697632,
'recall': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 0.0},
'tDistributionFalsePositives': {'unsampledValue': 362.0},
'tDistributionPrecision': {'unsampledValue': 0.24583333730697632},
'tDistributionRecall': {'unsampledValue': 1.0},
'tDistributionTrueNegatives': {'unsampledValue': 12.0},
'tDistributionTruePositives': {'unsampledValue': 118.0},
'threshold': 0.10000000149011612,
'trueNegatives': 12.0,
'truePositives': 118.0},
{'boundedFalseNegatives': {'value': 10.0},
'boundedFalsePositives': {'value': 289.0},
'boundedPrecision': {'value': 0.27204030752182007},
'boundedRecall': {'value': 0.9152542352676392},
'boundedTrueNegatives': {'value': 85.0},
'boundedTruePositives': {'value': 108.0},
'falseNegatives': 10.0,
'falsePositives': 289.0,
'precision': 0.27204030752182007,
'recall': 0.9152542352676392,
'tDistributionFalseNegatives': {'unsampledValue': 10.0},
'tDistributionFalsePositives': {'unsampledValue': 289.0},
'tDistributionPrecision': {'unsampledValue': 0.27204030752182007},
'tDistributionRecall': {'unsampledValue': 0.9152542352676392},
'tDistributionTrueNegatives': {'unsampledValue': 85.0},
'tDistributionTruePositives': {'unsampledValue': 108.0},
'threshold': 0.30000001192092896,
'trueNegatives': 85.0,
'truePositives': 108.0},
{'boundedFalseNegatives': {'value': 31.0},
'boundedFalsePositives': {'value': 199.0},
'boundedPrecision': {'value': 0.3041957914829254},
'boundedRecall': {'value': 0.7372881174087524},
'boundedTrueNegatives': {'value': 175.0},
'boundedTruePositives': {'value': 87.0},
'falseNegatives': 31.0,
'falsePositives': 199.0,
'precision': 0.3041957914829254,
'recall': 0.7372881174087524,
'tDistributionFalseNegatives': {'unsampledValue': 31.0},
'tDistributionFalsePositives': {'unsampledValue': 199.0},
'tDistributionPrecision': {'unsampledValue': 0.3041957914829254},
'tDistributionRecall': {'unsampledValue': 0.7372881174087524},
'tDistributionTrueNegatives': {'unsampledValue': 175.0},
'tDistributionTruePositives': {'unsampledValue': 87.0},
'threshold': 0.5,
'trueNegatives': 175.0,
'truePositives': 87.0},
{'boundedFalseNegatives': {'value': 58.0},
'boundedFalsePositives': {'value': 111.0},
'boundedPrecision': {'value': 0.35087719559669495},
'boundedRecall': {'value': 0.508474588394165},
'boundedTrueNegatives': {'value': 263.0},
'boundedTruePositives': {'value': 60.0},
'falseNegatives': 58.0,
'falsePositives': 111.0,
'precision': 0.35087719559669495,
'recall': 0.508474588394165,
'tDistributionFalseNegatives': {'unsampledValue': 58.0},
'tDistributionFalsePositives': {'unsampledValue': 111.0},
'tDistributionPrecision': {'unsampledValue': 0.35087719559669495},
'tDistributionRecall': {'unsampledValue': 0.508474588394165},
'tDistributionTrueNegatives': {'unsampledValue': 263.0},
'tDistributionTruePositives': {'unsampledValue': 60.0},
'threshold': 0.699999988079071,
'trueNegatives': 263.0,
'truePositives': 60.0},
{'boundedFalseNegatives': {'value': 97.0},
'boundedFalsePositives': {'value': 16.0},
'boundedPrecision': {'value': 0.5675675868988037},
'boundedRecall': {'value': 0.17796610295772552},
'boundedTrueNegatives': {'value': 358.0},
'boundedTruePositives': {'value': 21.0},
'falseNegatives': 97.0,
'falsePositives': 16.0,
'precision': 0.5675675868988037,
'recall': 0.17796610295772552,
'tDistributionFalseNegatives': {'unsampledValue': 97.0},
'tDistributionFalsePositives': {'unsampledValue': 16.0},
'tDistributionPrecision': {'unsampledValue': 0.5675675868988037},
'tDistributionRecall': {'unsampledValue': 0.17796610295772552},
'tDistributionTrueNegatives': {'unsampledValue': 358.0},
'tDistributionTruePositives': {'unsampledValue': 21.0},
'threshold': 0.8999999761581421,
'trueNegatives': 358.0,
'truePositives': 21.0}]} },
'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7541666626930237},
'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.7279596924781799},
'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6958041787147522},
'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.6491228342056274},
'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4324324429035187},
'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08474576473236084},
'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.26271185278892517},
'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.49152541160583496},
'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8220338821411133},
'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.10526315867900848},
'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.15048544108867645},
'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.180685356259346},
'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.2131868153810501},
'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9679144620895386},
'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7727272510528564},
'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.5320855379104614},
'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.2967914342880249},
'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.04278074949979782},
'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.024390242993831635},
'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.19308942556381226},
'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.4186991751194},
'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6524389982223511},
'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9247967600822449},
'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9756097793579102},
'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.8069105744361877},
'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5813007950782776},
'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.34756097197532654},
'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.07520325481891632},
'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.03208556026220322},
'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.22727273404598236},
'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.4679144322872162},
'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7032085657119751},
'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9572192430496216},
'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9152542352676392},
'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7372881174087524},
'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.508474588394165},
'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.17796610295772552},
'precision': {'doubleValue': 0.3041957914829254},
'prediction/mean': {'doubleValue': 0.5550199747085571},
'recall': {'doubleValue': 0.7372881174087524} }
Utilizzare get_metrics_for_all_slices() per ottenere le metriche per tutte le sezioni come un dizionario che associa ciascuna sezione al dizionario delle metriche corrispondente ottenuto eseguendo get_metrics_for_slice() su di esso.
pp.pprint(eval_result.get_metrics_for_all_slices())
{(): {'accuracy': {'doubleValue': 0.7191107273101807},
'accuracy_baseline': {'doubleValue': 0.9198060631752014},
'auc': {'doubleValue': 0.7972093224525452},
'auc_precision_recall': {'doubleValue': 0.3029831349849701},
'average_loss': {'doubleValue': 0.5623680949211121},
'label/mean': {'doubleValue': 0.08019392192363739},
'post_export_metrics/example_count': {'doubleValue': 721950.0},
'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 340.0},
'boundedFalsePositives': {'value': 612773.0},
'boundedPrecision': {'value': 0.08586231619119644},
'boundedRecall': {'value': 0.9941273927688599},
'boundedTrueNegatives': {'value': 51281.0},
'boundedTruePositives': {'value': 57556.0},
'falseNegatives': 340.0,
'falsePositives': 612773.0,
'precision': 0.08586231619119644,
'recall': 0.9941273927688599,
'tDistributionFalseNegatives': {'unsampledValue': 340.0},
'tDistributionFalsePositives': {'unsampledValue': 612773.0},
'tDistributionPrecision': {'unsampledValue': 0.08586231619119644},
'tDistributionRecall': {'unsampledValue': 0.9941273927688599},
'tDistributionTrueNegatives': {'unsampledValue': 51281.0},
'tDistributionTruePositives': {'unsampledValue': 57556.0},
'threshold': 0.10000000149011612,
'trueNegatives': 51281.0,
'truePositives': 57556.0},
{'boundedFalseNegatives': {'value': 4971.0},
'boundedFalsePositives': {'value': 385684.0},
'boundedPrecision': {'value': 0.12066555768251419},
'boundedRecall': {'value': 0.9141391515731812},
'boundedTrueNegatives': {'value': 278370.0},
'boundedTruePositives': {'value': 52925.0},
'falseNegatives': 4971.0,
'falsePositives': 385684.0,
'precision': 0.12066555768251419,
'recall': 0.9141391515731812,
'tDistributionFalseNegatives': {'unsampledValue': 4971.0},
'tDistributionFalsePositives': {'unsampledValue': 385684.0},
'tDistributionPrecision': {'unsampledValue': 0.12066555768251419},
'tDistributionRecall': {'unsampledValue': 0.9141391515731812},
'tDistributionTrueNegatives': {'unsampledValue': 278370.0},
'tDistributionTruePositives': {'unsampledValue': 52925.0},
'threshold': 0.30000001192092896,
'trueNegatives': 278370.0,
'truePositives': 52925.0},
{'boundedFalseNegatives': {'value': 15684.0},
'boundedFalsePositives': {'value': 187104.0},
'boundedPrecision': {'value': 0.18407785892486572},
'boundedRecall': {'value': 0.7291004657745361},
'boundedTrueNegatives': {'value': 476950.0},
'boundedTruePositives': {'value': 42212.0},
'falseNegatives': 15684.0,
'falsePositives': 187104.0,
'precision': 0.18407785892486572,
'recall': 0.7291004657745361,
'tDistributionFalseNegatives': {'unsampledValue': 15684.0},
'tDistributionFalsePositives': {'unsampledValue': 187104.0},
'tDistributionPrecision': {'unsampledValue': 0.18407785892486572},
'tDistributionRecall': {'unsampledValue': 0.7291004657745361},
'tDistributionTrueNegatives': {'unsampledValue': 476950.0},
'tDistributionTruePositives': {'unsampledValue': 42212.0},
'threshold': 0.5,
'trueNegatives': 476950.0,
'truePositives': 42212.0},
{'boundedFalseNegatives': {'value': 31392.0},
'boundedFalsePositives': {'value': 64497.0},
'boundedPrecision': {'value': 0.291249543428421},
'boundedRecall': {'value': 0.4577863812446594},
'boundedTrueNegatives': {'value': 599557.0},
'boundedTruePositives': {'value': 26504.0},
'falseNegatives': 31392.0,
'falsePositives': 64497.0,
'precision': 0.291249543428421,
'recall': 0.4577863812446594,
'tDistributionFalseNegatives': {'unsampledValue': 31392.0},
'tDistributionFalsePositives': {'unsampledValue': 64497.0},
'tDistributionPrecision': {'unsampledValue': 0.291249543428421},
'tDistributionRecall': {'unsampledValue': 0.4577863812446594},
'tDistributionTrueNegatives': {'unsampledValue': 599557.0},
'tDistributionTruePositives': {'unsampledValue': 26504.0},
'threshold': 0.699999988079071,
'trueNegatives': 599557.0,
'truePositives': 26504.0},
{'boundedFalseNegatives': {'value': 51089.0},
'boundedFalsePositives': {'value': 6354.0},
'boundedPrecision': {'value': 0.5172099471092224},
'boundedRecall': {'value': 0.11757288873195648},
'boundedTrueNegatives': {'value': 657700.0},
'boundedTruePositives': {'value': 6807.0},
'falseNegatives': 51089.0,
'falsePositives': 6354.0,
'precision': 0.5172099471092224,
'recall': 0.11757288873195648,
'tDistributionFalseNegatives': {'unsampledValue': 51089.0},
'tDistributionFalsePositives': {'unsampledValue': 6354.0},
'tDistributionPrecision': {'unsampledValue': 0.5172099471092224},
'tDistributionRecall': {'unsampledValue': 0.11757288873195648},
'tDistributionTrueNegatives': {'unsampledValue': 657700.0},
'tDistributionTruePositives': {'unsampledValue': 6807.0},
'threshold': 0.8999999761581421,
'trueNegatives': 657700.0,
'truePositives': 6807.0}]} },
'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.9141376614570618},
'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.8793344497680664},
'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.8159221410751343},
'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.7087504267692566},
'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4827900528907776},
'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0058725993148982525},
'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08586085587739944},
'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.27089953422546387},
'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.5422136187553406},
'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8824270963668823},
'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.006586466915905476},
'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.017544230446219444},
'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.03183702379465103},
'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.04975362494587898},
'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.07207927852869034},
'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9227758646011353},
'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.5808021426200867},
'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.28176021575927734},
'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.09712613373994827},
'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.009568499401211739},
'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.07150217890739441},
'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.3924662470817566},
'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.6823658347129822},
'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.8739510774612427},
'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9817702174186707},
'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9284977912902832},
'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.6075337529182434},
'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.3176341950893402},
'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.12604889273643494},
'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.01822979375720024},
'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.07722414284944534},
'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.41919782757759094},
'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.7182397842407227},
'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.9028738737106323},
'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9904314875602722},
'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 0.9941273927688599},
'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9141391515731812},
'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7291004657745361},
'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.4577863812446594},
'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.11757288873195648},
'precision': {'doubleValue': 0.18407785892486572},
'prediction/mean': {'doubleValue': 0.40011298656463623},
'recall': {'doubleValue': 0.7291004657745361} },
(('sexual_orientation', 'bisexual'),): {'accuracy': {'doubleValue': 0.5431034564971924},
'accuracy_baseline': {'doubleValue': 0.8017241358757019},
'auc': {'doubleValue': 0.6259934902191162},
'auc_precision_recall': {'doubleValue': 0.3277454376220703},
'average_loss': {'doubleValue': 0.7425483465194702},
'label/mean': {'doubleValue': 0.1982758641242981},
'post_export_metrics/example_count': {'doubleValue': 116.0},
'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0},
'boundedFalsePositives': {'value': 84.0},
'boundedPrecision': {'value': 0.21495327353477478},
'boundedRecall': {'value': 1.0},
'boundedTrueNegatives': {'value': 9.0},
'boundedTruePositives': {'value': 23.0},
'falsePositives': 84.0,
'precision': 0.21495327353477478,
'recall': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 0.0},
'tDistributionFalsePositives': {'unsampledValue': 84.0},
'tDistributionPrecision': {'unsampledValue': 0.21495327353477478},
'tDistributionRecall': {'unsampledValue': 1.0},
'tDistributionTrueNegatives': {'unsampledValue': 9.0},
'tDistributionTruePositives': {'unsampledValue': 23.0},
'threshold': 0.10000000149011612,
'trueNegatives': 9.0,
'truePositives': 23.0},
{'boundedFalseNegatives': {'value': 4.0},
'boundedFalsePositives': {'value': 67.0},
'boundedPrecision': {'value': 0.22093023359775543},
'boundedRecall': {'value': 0.8260869383811951},
'boundedTrueNegatives': {'value': 26.0},
'boundedTruePositives': {'value': 19.0},
'falseNegatives': 4.0,
'falsePositives': 67.0,
'precision': 0.22093023359775543,
'recall': 0.8260869383811951,
'tDistributionFalseNegatives': {'unsampledValue': 4.0},
'tDistributionFalsePositives': {'unsampledValue': 67.0},
'tDistributionPrecision': {'unsampledValue': 0.22093023359775543},
'tDistributionRecall': {'unsampledValue': 0.8260869383811951},
'tDistributionTrueNegatives': {'unsampledValue': 26.0},
'tDistributionTruePositives': {'unsampledValue': 19.0},
'threshold': 0.30000001192092896,
'trueNegatives': 26.0,
'truePositives': 19.0},
{'boundedFalseNegatives': {'value': 9.0},
'boundedFalsePositives': {'value': 44.0},
'boundedPrecision': {'value': 0.24137930572032928},
'boundedRecall': {'value': 0.6086956262588501},
'boundedTrueNegatives': {'value': 49.0},
'boundedTruePositives': {'value': 14.0},
'falseNegatives': 9.0,
'falsePositives': 44.0,
'precision': 0.24137930572032928,
'recall': 0.6086956262588501,
'tDistributionFalseNegatives': {'unsampledValue': 9.0},
'tDistributionFalsePositives': {'unsampledValue': 44.0},
'tDistributionPrecision': {'unsampledValue': 0.24137930572032928},
'tDistributionRecall': {'unsampledValue': 0.6086956262588501},
'tDistributionTrueNegatives': {'unsampledValue': 49.0},
'tDistributionTruePositives': {'unsampledValue': 14.0},
'threshold': 0.5,
'trueNegatives': 49.0,
'truePositives': 14.0},
{'boundedFalseNegatives': {'value': 16.0},
'boundedFalsePositives': {'value': 20.0},
'boundedPrecision': {'value': 0.25925925374031067},
'boundedRecall': {'value': 0.30434781312942505},
'boundedTrueNegatives': {'value': 73.0},
'boundedTruePositives': {'value': 7.0},
'falseNegatives': 16.0,
'falsePositives': 20.0,
'precision': 0.25925925374031067,
'recall': 0.30434781312942505,
'tDistributionFalseNegatives': {'unsampledValue': 16.0},
'tDistributionFalsePositives': {'unsampledValue': 20.0},
'tDistributionPrecision': {'unsampledValue': 0.25925925374031067},
'tDistributionRecall': {'unsampledValue': 0.30434781312942505},
'tDistributionTrueNegatives': {'unsampledValue': 73.0},
'tDistributionTruePositives': {'unsampledValue': 7.0},
'threshold': 0.699999988079071,
'trueNegatives': 73.0,
'truePositives': 7.0},
{'boundedFalseNegatives': {'value': 22.0},
'boundedFalsePositives': {'value': 0.0},
'boundedPrecision': {'value': 1.0},
'boundedRecall': {'value': 0.043478261679410934},
'boundedTrueNegatives': {'value': 93.0},
'boundedTruePositives': {'value': 1.0},
'falseNegatives': 22.0,
'precision': 1.0,
'recall': 0.043478261679410934,
'tDistributionFalseNegatives': {'unsampledValue': 22.0},
'tDistributionFalsePositives': {'unsampledValue': 0.0},
'tDistributionPrecision': {'unsampledValue': 1.0},
'tDistributionRecall': {'unsampledValue': 0.043478261679410934},
'tDistributionTrueNegatives': {'unsampledValue': 93.0},
'tDistributionTruePositives': {'unsampledValue': 1.0},
'threshold': 0.8999999761581421,
'trueNegatives': 93.0,
'truePositives': 1.0}]} },
'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7850467562675476},
'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.7790697813034058},
'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.7586206793785095},
'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.7407407164573669},
'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.17391304671764374},
'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.3913043439388275},
'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.695652186870575},
'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.95652174949646},
'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.13333334028720856},
'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.1551724076271057},
'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.17977528274059296},
'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.19130434095859528},
'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9032257795333862},
'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7204301357269287},
'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.47311827540397644},
'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.2150537669658661},
'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.0},
'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.07758620381355286},
'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.2586206793785095},
'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.5},
'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.767241358757019},
'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9913793206214905},
'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9224137663841248},
'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.7413793206214905},
'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5},
'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.23275862634181976},
'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.008620689623057842},
'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.09677419066429138},
'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.2795698940753937},
'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.5268816947937012},
'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7849462628364563},
'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.8260869383811951},
'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.6086956262588501},
'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.30434781312942505},
'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.043478261679410934},
'precision': {'doubleValue': 0.24137930572032928},
'prediction/mean': {'doubleValue': 0.4882037043571472},
'recall': {'doubleValue': 0.6086956262588501} },
(('sexual_orientation', 'heterosexual'),): {'accuracy': {'doubleValue': 0.5325203537940979},
'accuracy_baseline': {'doubleValue': 0.7601625919342041},
'auc': {'doubleValue': 0.6657300591468811},
'auc_precision_recall': {'doubleValue': 0.40228524804115295},
'average_loss': {'doubleValue': 0.8246824741363525},
'label/mean': {'doubleValue': 0.2398373931646347},
'post_export_metrics/example_count': {'doubleValue': 492.0},
'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0},
'boundedFalsePositives': {'value': 362.0},
'boundedPrecision': {'value': 0.24583333730697632},
'boundedRecall': {'value': 1.0},
'boundedTrueNegatives': {'value': 12.0},
'boundedTruePositives': {'value': 118.0},
'falsePositives': 362.0,
'precision': 0.24583333730697632,
'recall': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 0.0},
'tDistributionFalsePositives': {'unsampledValue': 362.0},
'tDistributionPrecision': {'unsampledValue': 0.24583333730697632},
'tDistributionRecall': {'unsampledValue': 1.0},
'tDistributionTrueNegatives': {'unsampledValue': 12.0},
'tDistributionTruePositives': {'unsampledValue': 118.0},
'threshold': 0.10000000149011612,
'trueNegatives': 12.0,
'truePositives': 118.0},
{'boundedFalseNegatives': {'value': 10.0},
'boundedFalsePositives': {'value': 289.0},
'boundedPrecision': {'value': 0.27204030752182007},
'boundedRecall': {'value': 0.9152542352676392},
'boundedTrueNegatives': {'value': 85.0},
'boundedTruePositives': {'value': 108.0},
'falseNegatives': 10.0,
'falsePositives': 289.0,
'precision': 0.27204030752182007,
'recall': 0.9152542352676392,
'tDistributionFalseNegatives': {'unsampledValue': 10.0},
'tDistributionFalsePositives': {'unsampledValue': 289.0},
'tDistributionPrecision': {'unsampledValue': 0.27204030752182007},
'tDistributionRecall': {'unsampledValue': 0.9152542352676392},
'tDistributionTrueNegatives': {'unsampledValue': 85.0},
'tDistributionTruePositives': {'unsampledValue': 108.0},
'threshold': 0.30000001192092896,
'trueNegatives': 85.0,
'truePositives': 108.0},
{'boundedFalseNegatives': {'value': 31.0},
'boundedFalsePositives': {'value': 199.0},
'boundedPrecision': {'value': 0.3041957914829254},
'boundedRecall': {'value': 0.7372881174087524},
'boundedTrueNegatives': {'value': 175.0},
'boundedTruePositives': {'value': 87.0},
'falseNegatives': 31.0,
'falsePositives': 199.0,
'precision': 0.3041957914829254,
'recall': 0.7372881174087524,
'tDistributionFalseNegatives': {'unsampledValue': 31.0},
'tDistributionFalsePositives': {'unsampledValue': 199.0},
'tDistributionPrecision': {'unsampledValue': 0.3041957914829254},
'tDistributionRecall': {'unsampledValue': 0.7372881174087524},
'tDistributionTrueNegatives': {'unsampledValue': 175.0},
'tDistributionTruePositives': {'unsampledValue': 87.0},
'threshold': 0.5,
'trueNegatives': 175.0,
'truePositives': 87.0},
{'boundedFalseNegatives': {'value': 58.0},
'boundedFalsePositives': {'value': 111.0},
'boundedPrecision': {'value': 0.35087719559669495},
'boundedRecall': {'value': 0.508474588394165},
'boundedTrueNegatives': {'value': 263.0},
'boundedTruePositives': {'value': 60.0},
'falseNegatives': 58.0,
'falsePositives': 111.0,
'precision': 0.35087719559669495,
'recall': 0.508474588394165,
'tDistributionFalseNegatives': {'unsampledValue': 58.0},
'tDistributionFalsePositives': {'unsampledValue': 111.0},
'tDistributionPrecision': {'unsampledValue': 0.35087719559669495},
'tDistributionRecall': {'unsampledValue': 0.508474588394165},
'tDistributionTrueNegatives': {'unsampledValue': 263.0},
'tDistributionTruePositives': {'unsampledValue': 60.0},
'threshold': 0.699999988079071,
'trueNegatives': 263.0,
'truePositives': 60.0},
{'boundedFalseNegatives': {'value': 97.0},
'boundedFalsePositives': {'value': 16.0},
'boundedPrecision': {'value': 0.5675675868988037},
'boundedRecall': {'value': 0.17796610295772552},
'boundedTrueNegatives': {'value': 358.0},
'boundedTruePositives': {'value': 21.0},
'falseNegatives': 97.0,
'falsePositives': 16.0,
'precision': 0.5675675868988037,
'recall': 0.17796610295772552,
'tDistributionFalseNegatives': {'unsampledValue': 97.0},
'tDistributionFalsePositives': {'unsampledValue': 16.0},
'tDistributionPrecision': {'unsampledValue': 0.5675675868988037},
'tDistributionRecall': {'unsampledValue': 0.17796610295772552},
'tDistributionTrueNegatives': {'unsampledValue': 358.0},
'tDistributionTruePositives': {'unsampledValue': 21.0},
'threshold': 0.8999999761581421,
'trueNegatives': 358.0,
'truePositives': 21.0}]} },
'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7541666626930237},
'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.7279596924781799},
'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6958041787147522},
'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.6491228342056274},
'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.4324324429035187},
'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.08474576473236084},
'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.26271185278892517},
'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.49152541160583496},
'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8220338821411133},
'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.10526315867900848},
'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.15048544108867645},
'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.180685356259346},
'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.2131868153810501},
'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9679144620895386},
'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7727272510528564},
'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.5320855379104614},
'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.2967914342880249},
'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.04278074949979782},
'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.024390242993831635},
'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.19308942556381226},
'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.4186991751194},
'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6524389982223511},
'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9247967600822449},
'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9756097793579102},
'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.8069105744361877},
'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5813007950782776},
'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.34756097197532654},
'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.07520325481891632},
'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.03208556026220322},
'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.22727273404598236},
'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.4679144322872162},
'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7032085657119751},
'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9572192430496216},
'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9152542352676392},
'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7372881174087524},
'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.508474588394165},
'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.17796610295772552},
'precision': {'doubleValue': 0.3041957914829254},
'prediction/mean': {'doubleValue': 0.5550199747085571},
'recall': {'doubleValue': 0.7372881174087524} },
(('sexual_orientation', 'homosexual_gay_or_lesbian'),): {'accuracy': {'doubleValue': 0.5879271030426025},
'accuracy_baseline': {'doubleValue': 0.7182232141494751},
'auc': {'doubleValue': 0.7076172232627869},
'auc_precision_recall': {'doubleValue': 0.4747510254383087},
'average_loss': {'doubleValue': 0.73641037940979},
'label/mean': {'doubleValue': 0.2817767560482025},
'post_export_metrics/example_count': {'doubleValue': 4390.0},
'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 2.0},
'boundedFalsePositives': {'value': 3031.0},
'boundedPrecision': {'value': 0.28949835896492004},
'boundedRecall': {'value': 0.9983831644058228},
'boundedTrueNegatives': {'value': 122.0},
'boundedTruePositives': {'value': 1235.0},
'falseNegatives': 2.0,
'falsePositives': 3031.0,
'precision': 0.28949835896492004,
'recall': 0.9983831644058228,
'tDistributionFalseNegatives': {'unsampledValue': 2.0},
'tDistributionFalsePositives': {'unsampledValue': 3031.0},
'tDistributionPrecision': {'unsampledValue': 0.28949835896492004},
'tDistributionRecall': {'unsampledValue': 0.9983831644058228},
'tDistributionTrueNegatives': {'unsampledValue': 122.0},
'tDistributionTruePositives': {'unsampledValue': 1235.0},
'threshold': 0.10000000149011612,
'trueNegatives': 122.0,
'truePositives': 1235.0},
{'boundedFalseNegatives': {'value': 76.0},
'boundedFalsePositives': {'value': 2368.0},
'boundedPrecision': {'value': 0.32898837327957153},
'boundedRecall': {'value': 0.9385610222816467},
'boundedTrueNegatives': {'value': 785.0},
'boundedTruePositives': {'value': 1161.0},
'falseNegatives': 76.0,
'falsePositives': 2368.0,
'precision': 0.32898837327957153,
'recall': 0.9385610222816467,
'tDistributionFalseNegatives': {'unsampledValue': 76.0},
'tDistributionFalsePositives': {'unsampledValue': 2368.0},
'tDistributionPrecision': {'unsampledValue': 0.32898837327957153},
'tDistributionRecall': {'unsampledValue': 0.9385610222816467},
'tDistributionTrueNegatives': {'unsampledValue': 785.0},
'tDistributionTruePositives': {'unsampledValue': 1161.0},
'threshold': 0.30000001192092896,
'trueNegatives': 785.0,
'truePositives': 1161.0},
{'boundedFalseNegatives': {'value': 281.0},
'boundedFalsePositives': {'value': 1528.0},
'boundedPrecision': {'value': 0.38486313819885254},
'boundedRecall': {'value': 0.7728375196456909},
'boundedTrueNegatives': {'value': 1625.0},
'boundedTruePositives': {'value': 956.0},
'falseNegatives': 281.0,
'falsePositives': 1528.0,
'precision': 0.38486313819885254,
'recall': 0.7728375196456909,
'tDistributionFalseNegatives': {'unsampledValue': 281.0},
'tDistributionFalsePositives': {'unsampledValue': 1528.0},
'tDistributionPrecision': {'unsampledValue': 0.38486313819885254},
'tDistributionRecall': {'unsampledValue': 0.7728375196456909},
'tDistributionTrueNegatives': {'unsampledValue': 1625.0},
'tDistributionTruePositives': {'unsampledValue': 956.0},
'threshold': 0.5,
'trueNegatives': 1625.0,
'truePositives': 956.0},
{'boundedFalseNegatives': {'value': 611.0},
'boundedFalsePositives': {'value': 731.0},
'boundedPrecision': {'value': 0.4613117277622223},
'boundedRecall': {'value': 0.5060630440711975},
'boundedTrueNegatives': {'value': 2422.0},
'boundedTruePositives': {'value': 626.0},
'falseNegatives': 611.0,
'falsePositives': 731.0,
'precision': 0.4613117277622223,
'recall': 0.5060630440711975,
'tDistributionFalseNegatives': {'unsampledValue': 611.0},
'tDistributionFalsePositives': {'unsampledValue': 731.0},
'tDistributionPrecision': {'unsampledValue': 0.4613117277622223},
'tDistributionRecall': {'unsampledValue': 0.5060630440711975},
'tDistributionTrueNegatives': {'unsampledValue': 2422.0},
'tDistributionTruePositives': {'unsampledValue': 626.0},
'threshold': 0.699999988079071,
'trueNegatives': 2422.0,
'truePositives': 626.0},
{'boundedFalseNegatives': {'value': 1072.0},
'boundedFalsePositives': {'value': 107.0},
'boundedPrecision': {'value': 0.6066176295280457},
'boundedRecall': {'value': 0.1333872228860855},
'boundedTrueNegatives': {'value': 3046.0},
'boundedTruePositives': {'value': 165.0},
'falseNegatives': 1072.0,
'falsePositives': 107.0,
'precision': 0.6066176295280457,
'recall': 0.1333872228860855,
'tDistributionFalseNegatives': {'unsampledValue': 1072.0},
'tDistributionFalsePositives': {'unsampledValue': 107.0},
'tDistributionPrecision': {'unsampledValue': 0.6066176295280457},
'tDistributionRecall': {'unsampledValue': 0.1333872228860855},
'tDistributionTrueNegatives': {'unsampledValue': 3046.0},
'tDistributionTruePositives': {'unsampledValue': 165.0},
'threshold': 0.8999999761581421,
'trueNegatives': 3046.0,
'truePositives': 165.0}]} },
'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.7105016112327576},
'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.6710116267204285},
'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6151368618011475},
'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.5386883020401001},
'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.39338234066963196},
'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.001616814872249961},
'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.061438966542482376},
'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.22716249525547028},
'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.4939369559288025},
'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 0.8666127920150757},
'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.016129031777381897},
'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.08826945722103119},
'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.14742916822433472},
'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.20145070552825928},
'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.2603205442428589},
'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 0.9613066911697388},
'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.7510307431221008},
'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.48461782932281494},
'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.23184269666671753},
'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.03393593430519104},
'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.028246013447642326},
'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.19612756371498108},
'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.43416857719421387},
'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6908884048461914},
'post_export_metrics/negative_rate@0.90': {'doubleValue': 0.9380410313606262},
'post_export_metrics/positive_rate@0.10': {'doubleValue': 0.9717540144920349},
'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.8038724660873413},
'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.5658314228057861},
'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.309111624956131},
'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.06195899844169617},
'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.03869330883026123},
'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.24896924197673798},
'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.5153821706771851},
'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.7681573033332825},
'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 0.9660640954971313},
'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 0.9983831644058228},
'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 0.9385610222816467},
'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 0.7728375196456909},
'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 0.5060630440711975},
'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.1333872228860855},
'precision': {'doubleValue': 0.38486313819885254},
'prediction/mean': {'doubleValue': 0.5426298975944519},
'recall': {'doubleValue': 0.7728375196456909} },
(('sexual_orientation', 'other_sexual_orientation'),): {'accuracy': {'doubleValue': 0.6000000238418579},
'accuracy_baseline': {'doubleValue': 0.800000011920929},
'auc': {'doubleValue': 1.0},
'auc_precision_recall': {'doubleValue': 1.0},
'average_loss': {'doubleValue': 0.7428082823753357},
'label/mean': {'doubleValue': 0.20000000298023224},
'post_export_metrics/example_count': {'doubleValue': 5.0},
'post_export_metrics/fairness/confusion_matrix_at_thresholds': {'confusionMatrixAtThresholds': {'matrices': [{'boundedFalseNegatives': {'value': 0.0},
'boundedFalsePositives': {'value': 4.0},
'boundedPrecision': {'value': 0.20000000298023224},
'boundedRecall': {'value': 1.0},
'boundedTrueNegatives': {'value': 0.0},
'boundedTruePositives': {'value': 1.0},
'falsePositives': 4.0,
'precision': 0.20000000298023224,
'recall': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 0.0},
'tDistributionFalsePositives': {'unsampledValue': 4.0},
'tDistributionPrecision': {'unsampledValue': 0.20000000298023224},
'tDistributionRecall': {'unsampledValue': 1.0},
'tDistributionTrueNegatives': {'unsampledValue': 0.0},
'tDistributionTruePositives': {'unsampledValue': 1.0},
'threshold': 0.10000000149011612,
'truePositives': 1.0},
{'boundedFalseNegatives': {'value': 0.0},
'boundedFalsePositives': {'value': 3.0},
'boundedPrecision': {'value': 0.25},
'boundedRecall': {'value': 1.0},
'boundedTrueNegatives': {'value': 1.0},
'boundedTruePositives': {'value': 1.0},
'falsePositives': 3.0,
'precision': 0.25,
'recall': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 0.0},
'tDistributionFalsePositives': {'unsampledValue': 3.0},
'tDistributionPrecision': {'unsampledValue': 0.25},
'tDistributionRecall': {'unsampledValue': 1.0},
'tDistributionTrueNegatives': {'unsampledValue': 1.0},
'tDistributionTruePositives': {'unsampledValue': 1.0},
'threshold': 0.30000001192092896,
'trueNegatives': 1.0,
'truePositives': 1.0},
{'boundedFalseNegatives': {'value': 0.0},
'boundedFalsePositives': {'value': 2.0},
'boundedPrecision': {'value': 0.3333333432674408},
'boundedRecall': {'value': 1.0},
'boundedTrueNegatives': {'value': 2.0},
'boundedTruePositives': {'value': 1.0},
'falsePositives': 2.0,
'precision': 0.3333333432674408,
'recall': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 0.0},
'tDistributionFalsePositives': {'unsampledValue': 2.0},
'tDistributionPrecision': {'unsampledValue': 0.3333333432674408},
'tDistributionRecall': {'unsampledValue': 1.0},
'tDistributionTrueNegatives': {'unsampledValue': 2.0},
'tDistributionTruePositives': {'unsampledValue': 1.0},
'threshold': 0.5,
'trueNegatives': 2.0,
'truePositives': 1.0},
{'boundedFalseNegatives': {'value': 0.0},
'boundedFalsePositives': {'value': 1.0},
'boundedPrecision': {'value': 0.5},
'boundedRecall': {'value': 1.0},
'boundedTrueNegatives': {'value': 3.0},
'boundedTruePositives': {'value': 1.0},
'falsePositives': 1.0,
'precision': 0.5,
'recall': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 0.0},
'tDistributionFalsePositives': {'unsampledValue': 1.0},
'tDistributionPrecision': {'unsampledValue': 0.5},
'tDistributionRecall': {'unsampledValue': 1.0},
'tDistributionTrueNegatives': {'unsampledValue': 3.0},
'tDistributionTruePositives': {'unsampledValue': 1.0},
'threshold': 0.699999988079071,
'trueNegatives': 3.0,
'truePositives': 1.0},
{'boundedFalseNegatives': {'value': 1.0},
'boundedFalsePositives': {'value': 0.0},
'boundedPrecision': {'value': 0.0},
'boundedRecall': {'value': 0.0},
'boundedTrueNegatives': {'value': 4.0},
'boundedTruePositives': {'value': 0.0},
'falseNegatives': 1.0,
'tDistributionFalseNegatives': {'unsampledValue': 1.0},
'tDistributionFalsePositives': {'unsampledValue': 0.0},
'tDistributionPrecision': {'unsampledValue': 0.0},
'tDistributionRecall': {'unsampledValue': 0.0},
'tDistributionTrueNegatives': {'unsampledValue': 4.0},
'tDistributionTruePositives': {'unsampledValue': 0.0},
'threshold': 0.8999999761581421,
'trueNegatives': 4.0}]} },
'post_export_metrics/false_discovery_rate@0.10': {'doubleValue': 0.800000011920929},
'post_export_metrics/false_discovery_rate@0.30': {'doubleValue': 0.75},
'post_export_metrics/false_discovery_rate@0.50': {'doubleValue': 0.6666666865348816},
'post_export_metrics/false_discovery_rate@0.70': {'doubleValue': 0.5},
'post_export_metrics/false_discovery_rate@0.90': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.30': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.50': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.70': {'doubleValue': 0.0},
'post_export_metrics/false_negative_rate@0.90': {'doubleValue': 1.0},
'post_export_metrics/false_omission_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/false_omission_rate@0.30': {'doubleValue': 0.0},
'post_export_metrics/false_omission_rate@0.50': {'doubleValue': 0.0},
'post_export_metrics/false_omission_rate@0.70': {'doubleValue': 0.0},
'post_export_metrics/false_omission_rate@0.90': {'doubleValue': 0.20000000298023224},
'post_export_metrics/false_positive_rate@0.10': {'doubleValue': 1.0},
'post_export_metrics/false_positive_rate@0.30': {'doubleValue': 0.75},
'post_export_metrics/false_positive_rate@0.50': {'doubleValue': 0.5},
'post_export_metrics/false_positive_rate@0.70': {'doubleValue': 0.25},
'post_export_metrics/false_positive_rate@0.90': {'doubleValue': 0.0},
'post_export_metrics/negative_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/negative_rate@0.30': {'doubleValue': 0.20000000298023224},
'post_export_metrics/negative_rate@0.50': {'doubleValue': 0.4000000059604645},
'post_export_metrics/negative_rate@0.70': {'doubleValue': 0.6000000238418579},
'post_export_metrics/negative_rate@0.90': {'doubleValue': 1.0},
'post_export_metrics/positive_rate@0.10': {'doubleValue': 1.0},
'post_export_metrics/positive_rate@0.30': {'doubleValue': 0.800000011920929},
'post_export_metrics/positive_rate@0.50': {'doubleValue': 0.6000000238418579},
'post_export_metrics/positive_rate@0.70': {'doubleValue': 0.4000000059604645},
'post_export_metrics/positive_rate@0.90': {'doubleValue': 0.0},
'post_export_metrics/true_negative_rate@0.10': {'doubleValue': 0.0},
'post_export_metrics/true_negative_rate@0.30': {'doubleValue': 0.25},
'post_export_metrics/true_negative_rate@0.50': {'doubleValue': 0.5},
'post_export_metrics/true_negative_rate@0.70': {'doubleValue': 0.75},
'post_export_metrics/true_negative_rate@0.90': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.10': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.30': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.50': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.70': {'doubleValue': 1.0},
'post_export_metrics/true_positive_rate@0.90': {'doubleValue': 0.0},
'precision': {'doubleValue': 0.3333333432674408},
'prediction/mean': {'doubleValue': 0.5955214500427246},
'recall': {'doubleValue': 1.0} } }