Model Card Toolkit Concepts

Model Card

Model Cards are machine learning documents that provide context and transparency into a model's development and performance. They can be used to share model metadata and metrics with researchers, developers, reporters, and more.

Some use cases of model cards include:

  • Facilitating the exchange of information between model builders and product developers.
  • Informing users of ML models to make better-informed decisions about how to use them (or how not to use them).
  • Providing model information required for effective public oversight and accountability.

Schema

The Model Card schema is a proto describing a model card's available fields. A JSON interface is also available. These objects can be interfaced with other systems for storage, analysis, or visualization.

Today, the Model Card schema is strictly enforced. In Model Card Toolkit 2.0, this schema restriction will be lifted.

Graphics

Model Card Toolkit automatically generates graphics for TFX datasets and evaluation results. Graphics can also be manually created using a tool like Matplotlib, and written to a ModelCard - see Model Card API for details.

In the Model Card schema, graphics are stored in the Graphic.image field, and are encoded as base64-encoded strings. The Model Card Toolkit can help with generating base64 images.

Model Card Toolkit

The Model Card Toolkit allows you to generate Model Card documents, as well as Proto and JSON objects, with a streamlined Python interface.

Model Card API

The Model Card Toolkit includes a Model Card API consisting of a Python class. Updates made to a Model Card Python object are written to a Model Card proto object.

Graphics

The model_card_toolkit.utils.graphics.figure_to_base64str() function can be used to convert graphics, such as Matplotlib figures, to base64 strings.

Model Card Documents

By default, the generated model card document is a HTML file based on default_template.html.jinja. However, you can also provide your own custom Jinja template. These templates files can be any text-based format (HTML, Markdown, LaTeX, etc.). A Markdown template is provided as an example.

TFX and MLMD Integration

The Model Card Toolkit integrates with the TensorFlow Extended and ML Metadata tools. A Metadata Store can be used during Model Card Toolkit initialization to pre-populate many model card fields and generate training and evaluation plots. See this guide for documentation, or this demonstration for a detailed example.

Artifacts used by MCT:

Executions used by MCT:

  • Trainer: used to populate model name and version