tfma.metrics.specs_from_metrics

Returns specs for tf.keras.metrics/losses or tfma.metrics classes.

tfma.metrics.specs_from_metrics(
    metrics: Optional[_MetricsOrLosses] = None,
    unweighted_metrics: Optional[_MetricsOrLosses] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    binarize: Optional[tfma.BinarizationOptions] = None,
    aggregate: Optional[tfma.AggregationOptions] = None,
    query_key: Optional[str] = None,
    include_example_count: Optional[bool] = None,
    include_weighted_example_count: Optional[bool] = None
) -> List[tfma.MetricsSpec]

Examples:

metrics_specs = specs_from_metrics( [ tf.keras.metrics.BinaryAccuracy(), tfma.metrics.AUC(), tfma.metrics.MeanLabel(), tfma.metrics.MeanPrediction() ... ], unweighted=[ tfma.metrics.Precision(), tfma.metrics.Recall() ])

metrics_specs = specs_from_metrics({ 'output1': [ tf.keras.metrics.BinaryAccuracy(), tfma.metrics.AUC(), tfma.metrics.MeanLabel(), tfma.metrics.MeanPrediction() ... ], 'output2': [ tfma.metrics.Precision(), tfma.metrics.Recall(), ] })

metrics List of tfma.metrics.Metric, tf.keras.metrics.Metric, or tf.keras.losses.Loss. For multi-output models a dict of dicts may be passed where the first dict is indexed by the output_name. Whether these metrics are weighted or not will be determined based on whether the ModelSpec associated with the metrics contains example weight key settings or not.
unweighted_metrics Same as metrics only these metrics will not be weighted by example_weight regardless of the example weight key settings.
model_names Optional model names (if multi-model evaluation).
output_names Optional output names (if multi-output models). If the metrics are a dict this should not be set.
output_weights Optional output weights for creating overall metric aggregated across outputs (if multi-output model). If a weight is not provided for an output, it's weight defaults to 0.0 (i.e. output ignored).
binarize Optional settings for binarizing multi-class/multi-label metrics.
aggregate Optional settings for aggregating multi-class/multi-label metrics.
query_key Optional query key for query/ranking based metrics.
include_example_count True to add example_count metric. Default is True.
include_weighted_example_count True to add weighted example_count metric. Default is True. A weighted example count will be added per output for multi-output models.

MetricsSpecs based on options provided. A separate spec is returned for weighted vs unweighted metrics. A separate spec is also returned for each output if a dict of metrics per output is passed.