![]() | ![]() | ![]() | ![]() |
Genel Bakış
Bu aktivitede, Wild veri kümesindeki Etiketli Yüzler üzerindeki FaceSSD tahminlerini keşfetmek için Adalet Göstergelerini kullanacaksınız. Adillik Göstergeleri, TensorFlow Model Analizinin üzerine inşa edilmiş ve ürün hatlarındaki adalet ölçütlerinin düzenli olarak değerlendirilmesini sağlayan bir araçlar paketidir.
Veri Kümesi Hakkında
Bu alıştırmada, FaceSSD tahmin veri kümesiyle, FaceSSD API tarafından oluşturulan yaklaşık 200 bin farklı görüntü tahmini ve temel gerçekle çalışacaksınız.
Araçlar Hakkında
TensorFlow Model Analizi , hem TensorFlow hem de TensorFlow olmayan makine öğrenimi modellerini değerlendirmek için bir kitaplıktır. Kullanıcıların, modellerini büyük miktarda veri üzerinde dağıtılmış bir şekilde değerlendirmelerine, grafik içi ve diğer ölçümleri farklı veri dilimlerinde hesaplamalarına ve not defterlerinde görselleştirmelerine olanak tanır.
TensorFlow Veri Doğrulama , verilerinizi analiz etmek için kullanabileceğiniz bir araçtır. Verilerinizdeki eksik değerler ve veri dengesizlikleri gibi Adalet eşitsizliklerine yol açabilecek potansiyel sorunları bulmak için kullanabilirsiniz.
Adillik Göstergeleri ile kullanıcılar şunları yapabilecektir:
- Model performansını, tanımlı kullanıcı grupları arasında dilimlenmiş olarak değerlendirin
- Birden fazla eşikte güven aralıkları ve değerlendirmeler içeren sonuçlar konusunda kendinizden emin olun
İçe aktarılıyor
Fairness_indicators kitaplığını yüklemek için aşağıdaki kodu çalıştırın. Bu paket, bu alıştırmada kullanacağımız araçları içerir. Yeniden Başlatma Runtime istenebilir, ancak gerekli değildir.
pip install -q -U pip==20.2
pip install fairness-indicators
import os
import tempfile
import apache_beam as beam
import numpy as np
import pandas as pd
from datetime import datetime
import tensorflow_hub as hub
import tensorflow as tf
import tensorflow_model_analysis as tfma
import tensorflow_data_validation as tfdv
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_predict as agnostic_predict
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_evaluate_graph
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_extractor
from witwidget.notebook.visualization import WitConfigBuilder
from witwidget.notebook.visualization import WitWidget
Verileri İndirin ve Anlayın
Wild'daki Etiketli Yüzler, çift eşleştirme olarak da bilinen, yüz doğrulama için herkese açık bir karşılaştırma veri kümesidir. LFW, web'den toplanan 13.000'den fazla yüz görüntüsünü içerir.
Belirli bir görüntüde bir yüz olup olmadığını tahmin etmek için bu veri kümesinde FaceSSD tahminlerini çalıştırdık. Bu Colab'da, farklı cinsiyet grupları için model performansı arasında önemli farklar olup olmadığını gözlemlemek için verileri cinsiyete göre dilimleyeceğiz.
Bir görüntüde birden fazla yüz varsa, cinsiyet "EKSİK" olarak etiketlenir.
Veri kümesini, kolaylık sağlamak için Google Cloud Platform'da barındırdık. Verileri GCP'den indirmek için aşağıdaki kodu çalıştırın, verilerin indirilmesi ve analiz edilmesi yaklaşık bir dakika sürer.
data_location = tf.keras.utils.get_file('lfw_dataset.tf', 'https://storage.googleapis.com/facessd_dataset/lfw_dataset.tfrecord')
stats = tfdv.generate_statistics_from_tfrecord(data_location=data_location)
tfdv.visualize_statistics(stats)
Downloading data from https://storage.googleapis.com/facessd_dataset/lfw_dataset.tfrecord 200835072/200828483 [==============================] - 1s 0us/step WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)` WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_data_validation/utils/stats_util.py:247: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: `tf.data.TFRecordDataset(path)`
Sabitleri Tanımlama
BASE_DIR = tempfile.gettempdir()
tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')
compute_confidence_intervals = True
slice_key = 'object/groundtruth/Gender'
label_key = 'object/groundtruth/face'
prediction_key = 'object/prediction/face'
feature_map = {
slice_key:
tf.io.FixedLenFeature([], tf.string, default_value=['none']),
label_key:
tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
prediction_key:
tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
}
TFMA için Model Agnostik Yapılandırma
model_agnostic_config = agnostic_predict.ModelAgnosticConfig(
label_keys=[label_key],
prediction_keys=[prediction_key],
feature_spec=feature_map)
model_agnostic_extractors = [
model_agnostic_extractor.ModelAgnosticExtractor(
model_agnostic_config=model_agnostic_config, desired_batch_size=3),
tfma.extractors.slice_key_extractor.SliceKeyExtractor(
[tfma.slicer.SingleSliceSpec(),
tfma.slicer.SingleSliceSpec(columns=[slice_key])])
]
Adalet Geri Aramaları ve Hesaplama Adalet Ölçütleri
# Helper class for counting examples in beam PCollection
class CountExamples(beam.CombineFn):
def __init__(self, message):
self.message = message
def create_accumulator(self):
return 0
def add_input(self, current_sum, element):
return current_sum + 1
def merge_accumulators(self, accumulators):
return sum(accumulators)
def extract_output(self, final_sum):
if final_sum:
print("%s: %d"%(self.message, final_sum))
metrics_callbacks = [
tfma.post_export_metrics.fairness_indicators(
thresholds=[0.1, 0.3, 0.5, 0.7, 0.9],
labels_key=label_key,
target_prediction_keys=[prediction_key]),
tfma.post_export_metrics.auc(
curve='PR',
labels_key=label_key,
target_prediction_keys=[prediction_key]),
]
eval_shared_model = tfma.types.EvalSharedModel(
add_metrics_callbacks=metrics_callbacks,
construct_fn=model_agnostic_evaluate_graph.make_construct_fn(
add_metrics_callbacks=metrics_callbacks,
config=model_agnostic_config))
with beam.Pipeline() as pipeline:
# Read data.
data = (
pipeline
| 'ReadData' >> beam.io.ReadFromTFRecord(data_location))
# Count all examples.
data_count = (
data | 'Count number of examples' >> beam.CombineGlobally(
CountExamples('Before filtering "Gender:MISSING"')))
# If there are more than one face in image, the gender feature is 'MISSING'
# and we are filtering that image out.
def filter_missing_gender(element):
example = tf.train.Example.FromString(element)
if example.features.feature[slice_key].bytes_list.value[0] != b'MISSING':
yield element
filtered_data = (
data
| 'Filter Missing Gender' >> beam.ParDo(filter_missing_gender))
# Count after filtering "Gender:MISSING".
filtered_data_count = (
filtered_data | 'Count number of examples after filtering'
>> beam.CombineGlobally(
CountExamples('After filtering "Gender:MISSING"')))
# Because LFW data set has always faces by default, we are adding
# labels as 1.0 for all images.
def add_face_groundtruth(element):
example = tf.train.Example.FromString(element)
example.features.feature[label_key].float_list.value[:] = [1.0]
yield example.SerializeToString()
final_data = (
filtered_data
| 'Add Face Groundtruth' >> beam.ParDo(add_face_groundtruth))
# Run TFMA.
_ = (
final_data
| 'ExtractEvaluateAndWriteResults' >>
tfma.ExtractEvaluateAndWriteResults(
eval_shared_model=eval_shared_model,
compute_confidence_intervals=compute_confidence_intervals,
output_path=tfma_eval_result_path,
extractors=model_agnostic_extractors))
eval_result = tfma.load_eval_result(output_path=tfma_eval_result_path)
WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring send_type hint: <class 'NoneType'> WARNING:apache_beam.typehints.typehints:Ignoring return_type hint: <class 'NoneType'> WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/post_export_metrics/post_export_metrics.py:178: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_model_analysis/post_export_metrics/post_export_metrics.py:178: auc (from tensorflow.python.ops.metrics_impl) is deprecated and will be removed in a future version. Instructions for updating: The value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead. Before filtering "Gender:MISSING": 13836 After filtering "Gender:MISSING": 11544 WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching:
Adillik Göstergelerini Oluştur
Adillik Göstergeleri gerecini dışa aktarılan değerlendirme sonuçlarıyla işleyin.
Aşağıda, seçilen metriklerdeki her bir veri diliminin performansını gösteren çubuk grafikler göreceksiniz. Görselleştirmenin üst kısmındaki açılır menüleri kullanarak temel karşılaştırma dilimini ve görüntülenen eşikleri ayarlayabilirsiniz.
Bu kullanım senaryosu için alakalı bir metrik, geri çağırma olarak da bilinen gerçek pozitif orandır. Doğru_ pozitif_ oran grafiğini seçmek için sol taraftaki seçiciyi kullanın. Bu metrik değerler, model kartında görüntülenen değerlerle eşleşir.
Bazı fotoğraflar için, fotoğraftaki kişi doğru bir şekilde açıklanamayacak kadar küçükse, cinsiyet erkek veya kadın yerine genç olarak etiketlenir.
widget_view.render_fairness_indicator(eval_result=eval_result,
slicing_column=slice_key)
FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'post_export…