|View source on GitHub|
Decompose the Brier score into uncertainty, resolution, and reliability.
tfp.experimental.substrates.numpy.stats.brier_decomposition( labels, logits, name=None )
[Proper scoring rules] measure the quality of probabilistic predictions;
any proper scoring rule admits a [unique decomposition] as
Score = Uncertainty - Resolution + Reliability, where:
Uncertainty, is a generalized entropy of the average predictive distribution; it can both be positive or negative.
Resolution, is a generalized variance of individual predictive distributions; it is always non-negative. Difference in predictions reveal information, that is why a larger resolution improves the predictive score.
Reliability, a measure of calibration of predictions against the true frequency of events. It is always non-negative and a lower value here indicates better calibration.
This method estimates the above decomposition for the case of the Brier
scoring rule for discrete outcomes. For this, we need to discretize the space
of probability distributions; we choose a simple partition of the space into
nlabels events: given a distribution
nlabels outcomes, the index
k for which
p_k > p_i for all
i != k determines the discretization
outcome; that is,
p in M_k, where
M_k is the set of all distributions for
p_k is the largest value among all probabilities.
The estimation error of each component is O(k/n), where n is the number
of instances and k is the number of labels. There may be an error of this
order when compared to
: Tilmann Gneiting, Adrian E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, Vol. 102, 2007. https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf : Jochen Broecker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, Vol. 135, 2009. https://rmets.onlinelibrary.wiley.com/doi/epdf/10.1002/qj.456
labels: Tensor, (n,), with tf.int32 or tf.int64 elements containing ground truth class labels in the range [0,nlabels].
logits: Tensor, (n, nlabels), with logits for n instances and nlabels.
strname prefixed to Ops created by this function.
uncertainty: Tensor, scalar, the uncertainty component of the decomposition.
resolution: Tensor, scalar, the resolution component of the decomposition.
reliability: Tensor, scalar, the reliability component of the decomposition.