As you tweak your model during development, you need to check whether your changes are improving your model. Just checking accuracy may not be enough. For example, if you have a classifier for a problem in which 95% of your instances are positive, you may be able to improve accuracy by simply always predicting positive, but you won't have a very robust classifier.
The goal of TensorFlow Model Analysis is to provide a mechanism for model evaluation in TFX. TensorFlow Model Analysis allows you to perform model evaluations in the TFX pipeline, and view resultant metrics and plots in a Jupyter notebook. Specifically, it can provide:
- Metrics computed on entire training and holdout dataset, as well as next-day evaluations
- Tracking metrics over time
- Model quality performance on different feature slices
- Model validation for ensuring that model's maintain consistent performance
Try our TFMA tutorial.
See the install and get_started guides for information and examples on how to get setup in a standlone pipeline. Recall that TFMA is also used within the Evaluator component in TFX so these resources will be useful for getting started in TFX as well.