# Responsible AI # Learn how to integrate Responsible AI practices into your ML workflow using TensorFlow TensorFlow is committed to helping make progress in the responsible development of AI by sharing a collection of resources and tools with the ML community. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/responsible-ai-hero.svg) ## What is Responsible AI? The development of AI is creating new opportunities to solve challenging, real-world problems. It is also raising new questions about the best way to build AI systems that benefit everyone. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/pillars/best-practices.svg) #### Recommended best practices for AI Designing AI systems should follow software development best practices while taking a human-centered approach to ML ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/pillars/fairness.svg) #### Fairness As the impact of AI increases across sectors and societies, it is critical to work towards systems that are fair and inclusive to everyone ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/pillars/interpretability.svg) #### Interpretability Understanding and trusting AI systems is important to ensuring they are working as intended ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/pillars/privacy.svg) #### Privacy Training models off of sensitive data needs privacy preserving safeguards ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/pillars/security.svg) #### Security Identifying potential threats can help keep AI systems safe and secure [Learn more about Google's Responsible AI Practices](https://ai.google/responsibilities/responsible-ai-practices/) ## Responsible AI in your ML workflow Responsible AI practices can be incorporated at every step of the ML workflow. Here are some key questions to consider at each stage. Define problem Construct and prepare data Build and train model Evaluate model Deploy and monitor ### Who is my ML system for? The way actual users experience your system is essential to assessing the true impact of its predictions, recommendations, and decisions. Make sure to get input from a diverse set of users early on in your development process. ### Am I using a representative dataset? Is your data sampled in a way that represents your users (e.g. will be used for all ages, but you only have training data from senior citizens) and the real-world setting (e.g. will be used year-round, but you only have training data from the summer)? ### Is there real-world/human bias in my data? Underlying biases in data can contribute to complex feedback loops that reinforce existing stereotypes. ### What methods should I use to train my model? Use training methods that build fairness, interpretability, privacy, and security into the model. ### How is my model performing? Evaluate user experience in real-world scenarios across a broad spectrum of users, use cases, and contexts of use. Test and iterate in dogfood first, followed by continued testing after launch. ### Are there complex feedback loops? Even if everything in the overall system design is carefully crafted, ML-based models rarely operate with 100% perfection when applied to real, live data. When an issue occurs in a live product, consider whether it aligns with any existing societal disadvantages, and how it will be impacted by both short- and long-term solutions. ## Responsible AI tools for TensorFlow The TensorFlow ecosystem has a suite of tools and resources to help tackle some of the questions above. Step 1 ## Define problem Use the following resources to design models with Responsible AI in mind. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_pair-guidebook.jpg) [People + AI Research (PAIR) Guidebook](https://pair.withgoogle.com/guidebook/) Learn more about the AI development process and key considerations. [Learn more](https://pair.withgoogle.com/guidebook/) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_explorable-bias.jpg) [PAIR Explorables](https://pair.withgoogle.com/explorables/) Explore, via interactive visualizations, key questions and concepts in the realm of Responsible AI. [Learn more](https://pair.withgoogle.com/explorables/) Step 2 ## Construct and prepare data Use the following tools to examine data for potential biases. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_know-your-data.jpg) [Know Your Data (Beta)](https://knowyourdata.withgoogle.com/) Interactively investigate your dataset to improve data quality and mitigate fairness and bias issues. [Learn more](https://knowyourdata.withgoogle.com/) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_tensorflow-data-validation.jpg) [TF Data Validation](https://www.tensorflow.org/tfx/guide/tfdv) Analyze and transform data to detect problems and engineer more effective feature sets. [Learn more](https://www.tensorflow.org/tfx/guide/tfdv) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_data-cards.png) [Data Cards](https://research.google/static/documents/datasets/crowdsourced-high-quality-colombian-spanish-es-co-multi-speaker-speech-dataset.pdf) Create a transparency report for your dataset. [Learn more](https://research.google/static/documents/datasets/crowdsourced-high-quality-colombian-spanish-es-co-multi-speaker-speech-dataset.pdf) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/title_responsible-ai.jpg) [Monk Skin Tone Scale (MST)](https://www.skintone.google/) A more inclusive skin tone scale, open licensed, to make your data collection and model building needs more robust and inclusive. [Learn more](https://www.skintone.google/) Step 3 ## Build and train model Use the following tools to train models using privacy-preserving, interpretable techniques, and more. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_tensorflow-model-remediation-library.jpeg) [TF Model Remediation](https://www.tensorflow.org/responsible_ai/model_remediation) Train machine learning models to promote more equitable outcomes. [Learn more](https://www.tensorflow.org/responsible_ai/model_remediation) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_tensorflow-privacy.jpg) [TF Privacy](https://www.tensorflow.org/responsible_ai/privacy/guide) Train machine learning models with privacy. [Learn more](https://www.tensorflow.org/responsible_ai/privacy/guide) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_tensorflow-federated.jpg) [TF Federated](https://www.tensorflow.org/federated) Train machine learning models using federated learning techniques. [Learn more](https://www.tensorflow.org/federated) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_constrained-optimization.jpg) [TF Constrained Optimization](https://github.com/google-research/tensorflow_constrained_optimization/blob/master/README.md) Optimize inequality-constrained problems. [Learn more](https://github.com/google-research/tensorflow_constrained_optimization/blob/master/README.md) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_lattice.jpg) [TF Lattice](https://www.tensorflow.org/lattice/overview) Implement flexible, controlled, and interpretable lattice-based models. [Learn more](https://www.tensorflow.org/lattice/overview) Step 4 ## Evaluate model Debug, evaluate, and visualize model performance using the following tools. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_fairness-indicators.jpg) [Fairness Indicators](https://www.tensorflow.org/responsible_ai/fairness_indicators/guide) Evaluate commonly-identified fairness metrics for binary and multi-class classifiers. [Learn more](https://www.tensorflow.org/responsible_ai/fairness_indicators/guide) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_tensorflow-model-analysis.jpg) [TF Model Analysis](https://www.tensorflow.org/tfx/model_analysis/install) Evaluate models in a distributed manner and compute over different slices of data. [Learn more](https://www.tensorflow.org/tfx/model_analysis/install) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_what-if-tool.jpg) [What-If Tool](https://pair-code.github.io/what-if-tool/) Examine, evaluate, and compare machine learning models. [Learn more](https://pair-code.github.io/what-if-tool/) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_language-interpretability-tool.png) [Language Interpretability Tool](https://pair-code.github.io/lit/) Visualize and understand NLP models. [Learn more](https://pair-code.github.io/lit/) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_explainable-ai.jpg) [Explainable AI](https://cloud.google.com/explainable-ai) Develop interpretable and inclusive machine learning models. [Learn more](https://cloud.google.com/explainable-ai) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_tf-privacy-testing.png) [TF Privacy Tests](https://blog.tensorflow.org/2020/06/introducing-new-privacy-testing-library.html) Assess the privacy properties of classification models. [Learn more](https://blog.tensorflow.org/2020/06/introducing-new-privacy-testing-library.html) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_tensorboard.png) [TensorBoard](https://www.tensorflow.org/tensorboard/get_started) Measure and visualize the machine learning workflow. [Learn more](https://www.tensorflow.org/tensorboard/get_started) Step 5 ## Deploy and monitor Use the following tools to track and communicate about model context and details. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_model_card_toolkit.png) [Model Card Toolkit](https://www.tensorflow.org/responsible_ai/model_card_toolkit/guide) Generate model cards with ease using the Model Card toolkit. [Learn more](https://www.tensorflow.org/responsible_ai/model_card_toolkit/guide) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_ml-metadata.jpg) [ML Metadata](https://www.tensorflow.org/tfx/guide/mlmd) Record and retrieve metadata associated with ML developer and data scientist workflows. [Learn more](https://www.tensorflow.org/tfx/guide/mlmd) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/tools/tile_model-cards.jpg) [Model Cards](https://modelcards.withgoogle.com/about) Organize the essential facts of machine learning in a structured way. [Learn more](https://modelcards.withgoogle.com/about) ## Community resources Learn what the community is doing and explore ways to get involved. ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/other/tile_crowdsource.jpg) [Crowdsource by Google](https://crowdsource.google.com/) Help Google's products become more inclusive and representative of your language, region and culture. [Learn more](https://crowdsource.google.com/) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/other/tile_tensorflow.jpg) [Responsible AI DevPost Challenge](https://responsible-ai.devpost.com/) We asked participants to use TensorFlow 2.2 to build a model or application with Responsible AI principles in mind. Check out the gallery to see the winners and other amazing projects. [Learn more](https://responsible-ai.devpost.com/) ![](https://www.tensorflow.org/static/site-assets/images/marketing/resources/responsible-ai/other/tile_tf-dev-summit.jpg) Responsible AI with TensorFlow (TF Dev Summit '20) Introducing a framework to think about ML, fairness and privacy. Watch the video [Video](https://www.youtube.com/watch?v=UEECKh6PLhI) ## Explore Google AI resources to guide your AI/ML journey [See AI principles](https://ai.google/responsibilities/#our-principles)