tfma.metrics.to_label_prediction_example_weight

View source on GitHub

Yields label, prediction, and example weights for use in calculations.

Where applicable this function will perform model and output name lookups as well as any required class ID, top K, etc conversions. It will also apply prediction keys and label vocabularies given the necessary information is provided as part of the EvalConfig (or standard estimator based naming is used). The sparseness of labels will be inferred from the shapes of the labels and predictions (i.e. if the shapes are different then the labels will be assumed to be sparse).

If successful, the final output of calling this function will be a tuple of numpy arrays representing the label, prediction, and example weight respectively. Labels and predictions will be returned in the same shape provided (default behavior) unless (1) flatten is True in which case a series of values (one per class ID) will be returned with last dimension of size 1 or (2) a sub_key is used in which case the last dimension will be re-shaped to match the number of outputs selected (1 or top_k). Examples:

# default behavior # # Binary classification Input : labels=[1] predictions=[0.6] Output : (np.array([1]), np.array([0.6]), np.array([1.0])) # Multi-class classification w/ sparse labels Input : labels=[2] predictions=[0.3, 0.6, 0.1] Output: (np.array([2]), np.array([0.3, 0.6, 0.1]), np.array([1.0])) # Multi-class / multi-label classification w/ dense labels Input : labels=[0, 1, 1] predictions=[0.3, 0.6, 0.1] Output : (np.array([0, 1, 1]), np.array([0.3, 0.6, 0.1]), np.array([1.0]))

# flatten=True # # Multi-class classification w/ sparse labels Input : labels=[2], predictions=[0.3, 0.6, 0.1] Output : (np.array([0]), np.array([0.3]), np.array([1.0])), (np.array([0]), np.array([0.6]), np.array([1.0])), (np.array([1]), np.array([0.1]), np.array([1.0])) # Multi-class/multi-label classification w/ dense labels Input : labels=[0, 0, 1], predictions=[0.3, 0.6, 0.1] Output : (np.array([0]), np.array([0.3]), np.array([1.0])), (np.array([0]), np.array([0.6]), np.array([1.0])), (np.array([1]), np.array([0.1]), np.array([1.0]))

# sub_key.class_id=[2] # # Multi-class classification w/ sparse labels Input : labels=[2] predictions=[0.3, 0.6, 0.1] Output : (np.array([1]), np.array([0.1]), np.array([1.0])) # Multi-class classification w/ dense labels Input : labels=[0, 0, 1] predictions=[0.3, 0.6, 0.1] Output : (np.array([1]), np.array([0.1]), np.array([1.0]))

inputs Standard metric inputs.
eval_config Eval config
model_name Optional model name (if multi-model evaluation).
output_name Optional output name (if multi-output model type).
sub_key Optional sub key.
class_weights Optional class weights to apply to multi-class / multi-label labels and predictions. If used, flatten must also be True.
flatten True to flatten the final label and prediction outputs so that the yielded values are always arrays of size 1. For example, multi-class / multi-label outputs would be converted into label and prediction pairs that could then be processed by a binary classification metric in order to compute a micro average over all classes.
squeeze True to squeeze any outputs that have rank > 1. This transforms outputs such as np.array([[1]]) to np.array([1]).
allow_none True to allow labels or predictions with None values to be returned. The example weight will always be non-None.

Yields:

Tuple of (label, prediction, example_weight).