FlipRate is the rate at which predictions between models switch.

Inherits From: Metric

Given a pair of models and a threshold for converting continuous model outputs into boolean predictions, this metric will produce three numbers (keyed by separate MetricKeys):

  • (symmetric) flip rate: The number of times the boolean predictions don't match, regardless of the direction of the flip.
  • negative-to-positive flip rate: The rate at which the baseline model's boolean prediction is negative but the candidate model's is positive.
  • positive-to-negative flip rate: The rate at which the baseline model's boolean prediction is positive but the candidate model's is negative.

threshold The threshold to use for converting the model prediction into a boolean value that can be used for comparison between models.
flip_rate_name Metric name for symmetric flip rate.
neg_to_neg_flip_rate_name Metric name for the negative-to-negative flip rate.
neg_to_pos_flip_rate_name Metric name for the negative-to-positive flip rate.
pos_to_neg_flip_rate_name Metric name for the positive-to-negative flip rate.
pos_to_pos_flip_rate_name Metric name for the positive-to-positive flip rate.

compute_confidence_interval Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.



View source

Creates computations associated with metric.


View source


View source

Returns serializable config.