Model Card
Model Cards are machine learning documents that provide context and transparency into a model's development and performance. They can be used to share model metadata and metrics with researchers, developers, reporters, and more.
Some use cases of model cards include:
- Facilitating the exchange of information between model builders and product developers.
- Informing users of ML models to make better-informed decisions about how to use them (or how not to use them).
- Providing model information required for effective public oversight and accountability.
Schema
The Model Card schema is a proto describing a model card's available fields. A JSON interface is also available. These objects can be interfaced with other systems for storage, analysis, or visualization.
Today, the Model Card schema is strictly enforced. In Model Card Toolkit 2.0, this schema restriction will be lifted.
Graphics
Model Card Toolkit automatically generates graphics for TFX datasets and evaluation results. Graphics can also be manually created using a tool like Matplotlib, and written to a ModelCard - see Model Card API for details.
In the Model Card schema, graphics are stored in the Graphic.image field, and are encoded as base64-encoded strings. The Model Card Toolkit can help with generating base64 images.
Model Card Toolkit
The Model Card Toolkit allows you to generate Model Card documents, as well as Proto and JSON objects, with a streamlined Python interface.
Model Card API
The Model Card Toolkit includes a Model Card API consisting of a Python class. Updates made to a Model Card Python object are written to a Model Card proto object.
Graphics
The model_card_toolkit.utils.graphics.figure_to_base64str() function can be
used to convert graphics, such as Matplotlib figures, to base64 strings.
Model Card Documents
By default, the generated model card document is a HTML file based on default_template.html.jinja. However, you can also provide your own custom Jinja template. These templates files can be any text-based format (HTML, Markdown, LaTeX, etc.). A Markdown template is provided as an example.
TFX and MLMD Integration
The Model Card Toolkit integrates with the TensorFlow Extended and ML Metadata tools. A Metadata Store can be used during Model Card Toolkit initialization to pre-populate many model card fields and generate training and evaluation plots. See this guide for documentation, or this demonstration for a detailed example.
Artifacts used by MCT:
- Examples and ExampleStatistics: used to plot slice count graphs for each dataset (TFDV).
- Model and ModelEvaluation: used to plot TFMA sliced evaluation metrics.
Executions used by MCT:
- Trainer: used to populate model name and version