 Lihat di TensorFlow.org |  Jalankan di Google Colab |  Lihat di GitHub |  Unduh buku catatan |
Ringkasan
Dalam kegiatan ini, Anda akan menggunakan Keadilan Indikator untuk mengeksplorasi prediksi FaceSSD di Wajah Berlabel di dataset liar . Keadilan Indikator adalah seperangkat alat yang dibangun di atas Analisis Model TensorFlow yang memungkinkan evaluasi secara berkala metrik keadilan dalam pipa produk.
Tentang Kumpulan Data
Dalam latihan ini, Anda akan bekerja dengan kumpulan data prediksi FaceSSD, sekitar 200 ribu prediksi gambar berbeda dan groundtruth yang dihasilkan oleh FaceSSD API.
Tentang Alat
Analisis Model TensorFlow adalah library untuk mengevaluasi baik TensorFlow dan non-TensorFlow model pembelajaran mesin. Hal ini memungkinkan pengguna untuk mengevaluasi model mereka pada sejumlah besar data secara terdistribusi, menghitung dalam grafik dan metrik lainnya pada irisan data yang berbeda dan memvisualisasikannya di notebook.
TensorFlow Validasi Data adalah salah satu alat yang dapat digunakan untuk menganalisis data Anda. Anda dapat menggunakannya untuk menemukan potensi masalah dalam data Anda, seperti nilai yang hilang dan ketidakseimbangan data, yang dapat menyebabkan disparitas Kewajaran.
Dengan Keadilan Indikator , pengguna akan dapat:
- Evaluasi kinerja model, diiris di seluruh kelompok pengguna yang ditentukan
- Merasa yakin tentang hasil dengan interval kepercayaan dan evaluasi di berbagai ambang batas
Pengimporan
Jalankan kode berikut untuk menginstal perpustakaan fairness_indicators. Paket ini berisi alat yang akan kita gunakan dalam latihan ini. Restart Runtime mungkin diminta tetapi tidak diperlukan.
pip install -q -U pip==20.2pip install fairness-indicators
import os
import tempfile
import apache_beam as beam
import numpy as np
import pandas as pd
from datetime import datetime
import tensorflow_hub as hub
import tensorflow as tf
import tensorflow_model_analysis as tfma
import tensorflow_data_validation as tfdv
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_predict as agnostic_predict
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_evaluate_graph
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_extractor
from witwidget.notebook.visualization import WitConfigBuilder
from witwidget.notebook.visualization import WitWidget
Unduh dan Pahami Datanya
Wajah berlabel di Wild adalah dataset patokan umum untuk verifikasi wajah, juga dikenal sebagai pasangan yang cocok. LFW berisi lebih dari 13.000 gambar wajah yang dikumpulkan dari web.
Kami menjalankan prediksi FaceSSD pada kumpulan data ini untuk memprediksi apakah wajah ada dalam gambar tertentu. Dalam Colab ini, kami akan mengiris data menurut gender untuk mengamati apakah ada perbedaan yang signifikan antara performa model untuk kelompok gender yang berbeda.
Jika ada lebih dari satu wajah dalam sebuah gambar, jenis kelamin diberi label sebagai "HILANG".
Kami telah menghosting kumpulan data di Google Cloud Platform untuk kenyamanan. Jalankan kode berikut untuk mengunduh data dari GCP, data akan memakan waktu sekitar satu menit untuk diunduh dan dianalisis.
data_location = tf.keras.utils.get_file('lfw_dataset.tf', 'https://storage.googleapis.com/facessd_dataset/lfw_dataset.tfrecord')
stats = tfdv.generate_statistics_from_tfrecord(data_location=data_location)
tfdv.visualize_statistics(stats)
Downloading data from https://storage.googleapis.com/facessd_dataset/lfw_dataset.tfrecord 200835072/200828483 [==============================] - 1s 0us/step WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
Mendefinisikan Konstanta
BASE_DIR = tempfile.gettempdir()
tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')
compute_confidence_intervals = True
slice_key = 'object/groundtruth/Gender'
label_key = 'object/groundtruth/face'
prediction_key = 'object/prediction/face'
feature_map = {
slice_key:
tf.io.FixedLenFeature([], tf.string, default_value=['none']),
label_key:
tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
prediction_key:
tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
}
Konfigurasi Model Agnostik untuk TFMA
model_agnostic_config = agnostic_predict.ModelAgnosticConfig(
label_keys=[label_key],
prediction_keys=[prediction_key],
feature_spec=feature_map)
model_agnostic_extractors = [
model_agnostic_extractor.ModelAgnosticExtractor(
model_agnostic_config=model_agnostic_config, desired_batch_size=3),
tfma.extractors.slice_key_extractor.SliceKeyExtractor(
[tfma.slicer.SingleSliceSpec(),
tfma.slicer.SingleSliceSpec(columns=[slice_key])])
]
Panggilan Balik Kewajaran dan Metrik Kewajaran Komputasi
# Helper class for counting examples in beam PCollection
class CountExamples(beam.CombineFn):
def __init__(self, message):
self.message = message
def create_accumulator(self):
return 0
def add_input(self, current_sum, element):
return current_sum + 1
def merge_accumulators(self, accumulators):
return sum(accumulators)
def extract_output(self, final_sum):
if final_sum:
print("%s: %d"%(self.message, final_sum))
metrics_callbacks = [
tfma.post_export_metrics.fairness_indicators(
thresholds=[0.1, 0.3, 0.5, 0.7, 0.9],
labels_key=label_key,
target_prediction_keys=[prediction_key]),
tfma.post_export_metrics.auc(
curve='PR',
labels_key=label_key,
target_prediction_keys=[prediction_key]),
]
eval_shared_model = tfma.types.EvalSharedModel(
add_metrics_callbacks=metrics_callbacks,
construct_fn=model_agnostic_evaluate_graph.make_construct_fn(
add_metrics_callbacks=metrics_callbacks,
config=model_agnostic_config))
with beam.Pipeline() as pipeline:
# Read data.
data = (
pipeline
| 'ReadData' >> beam.io.ReadFromTFRecord(data_location))
# Count all examples.
data_count = (
data | 'Count number of examples' >> beam.CombineGlobally(
CountExamples('Before filtering "Gender:MISSING"')))
# If there are more than one face in image, the gender feature is 'MISSING'
# and we are filtering that image out.
def filter_missing_gender(element):
example = tf.train.Example.FromString(element)
if example.features.feature[slice_key].bytes_list.value[0] != b'MISSING':
yield element
filtered_data = (
data
| 'Filter Missing Gender' >> beam.ParDo(filter_missing_gender))
# Count after filtering "Gender:MISSING".
filtered_data_count = (
filtered_data | 'Count number of examples after filtering'
>> beam.CombineGlobally(
CountExamples('After filtering "Gender:MISSING"')))
# Because LFW data set has always faces by default, we are adding
# labels as 1.0 for all images.
def add_face_groundtruth(element):
example = tf.train.Example.FromString(element)
example.features.feature[label_key].float_list.value[:] = [1.0]
yield example.SerializeToString()
final_data = (
filtered_data
| 'Add Face Groundtruth' >> beam.ParDo(add_face_groundtruth))
# Run TFMA.
_ = (
final_data
| 'ExtractEvaluateAndWriteResults' >>
tfma.ExtractEvaluateAndWriteResults(
eval_shared_model=eval_shared_model,
compute_confidence_intervals=compute_confidence_intervals,
output_path=tfma_eval_result_path,
extractors=model_agnostic_extractors))
eval_result = tfma.load_eval_result(output_path=tfma_eval_result_path)
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/post_export_metrics/post_export_metrics.py:178: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/post_export_metrics/post_export_metrics.py:178: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. Before filtering "Gender:MISSING": 13836 After filtering "Gender:MISSING": 11544 WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching:
Render Indikator Kewajaran
Render widget Indikator Kewajaran dengan hasil evaluasi yang diekspor.
Di bawah ini Anda akan melihat diagram batang yang menampilkan kinerja setiap potongan data pada metrik yang dipilih. Anda dapat menyesuaikan irisan perbandingan dasar serta ambang yang ditampilkan menggunakan menu tarik-turun di bagian atas visualisasi.
Metrik yang relevan untuk kasus penggunaan ini adalah true positive rate, juga dikenal sebagai recall. Gunakan pemilih di sisi kiri untuk memilih grafik untuk true_positive_rate. Nilai-nilai metrik sesuai dengan nilai-nilai yang ditampilkan pada kartu model .
Untuk beberapa foto, jenis kelamin diberi label sebagai muda, bukan laki-laki atau perempuan, jika orang dalam foto terlalu muda untuk dijelaskan secara akurat.
widget_view.render_fairness_indicator(eval_result=eval_result,
slicing_column=slice_key)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…