tfma.metrics.BooleanFlipRates

FlipRate is the rate at which predictions between models switch.

Inherits From: Metric

tfma.metrics.BooleanFlipRates(
    threshold: float = _DEFAULT_FLIP_RATE_THRESHOLD,
    flip_rate_name: str = FLIP_RATE_NAME,
    neg_to_neg_flip_rate_name: str = NEG_TO_NEG_FLIP_RATE_NAME,
    neg_to_pos_flip_rate_name: str = NEG_TO_POS_FLIP_RATE_NAME,
    pos_to_neg_flip_rate_name: str = POS_TO_NEG_FLIP_RATE_NAME,
    pos_to_pos_flip_rate_name: str = POS_TO_POS_FLIP_RATE_NAME
)

Given a pair of models and a threshold for converting continuous model outputs into boolean predictions, this metric will produce three numbers (keyed by separate MetricKeys):

  • (symmetric) flip rate: The number of times the boolean predictions don't match, regardless of the direction of the flip.
  • negative-to-positive flip rate: The rate at which the baseline model's boolean prediction is negative but the candidate model's is positive.
  • positive-to-negative flip rate: The rate at which the baseline model's boolean prediction is positive but the candidate model's is negative.

threshold The threshold to use for converting the model prediction into a boolean value that can be used for comparison between models.
flip_rate_name Metric name for symmetric flip rate.
neg_to_neg_flip_rate_name Metric name for the negative-to-negative flip rate.
neg_to_pos_flip_rate_name Metric name for the negative-to-positive flip rate.
pos_to_neg_flip_rate_name Metric name for the positive-to-negative flip rate.
pos_to_pos_flip_rate_name Metric name for the positive-to-positive flip rate.

compute_confidence_interval Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

Methods

computations

View source

computations(
    eval_config: Optional[tfma.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None
) -> tfma.metrics.MetricComputations

Creates computations associated with metric.

from_config

View source

@classmethod
from_config(
    config: Dict[str, Any]
) -> 'Metric'

get_config

View source

get_config() -> Dict[str, Any]

Returns serializable config.