<br />

|---|---|---|---|---|
| [![](https://www.tensorflow.org/images/tf_logo_32px.png)View on TensorFlow.org](https://www.tensorflow.org/hub/tutorials/bert_experts) | [![](https://www.tensorflow.org/images/colab_logo_32px.png)Run in Google Colab](https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/hub/tutorials/bert_experts.ipynb) | [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png)View on GitHub](https://github.com/tensorflow/docs/blob/master/site/en/hub/tutorials/bert_experts.ipynb) | [![](https://www.tensorflow.org/images/download_logo_32px.png)Download notebook](https://storage.googleapis.com/tensorflow_docs/docs/site/en/hub/tutorials/bert_experts.ipynb) | [![](https://www.tensorflow.org/images/hub_logo_32px.png)See TF Hub models](https://tfhub.dev/s?q=experts/bert) |

This colab demonstrates how to:

- Load BERT models from [TensorFlow Hub](https://tfhub.dev) that have been trained on different tasks including MNLI, SQuAD, and PubMed
- Use a matching preprocessing model to tokenize raw text and convert it to ids
- Generate the pooled and sequence output from the token input ids using the loaded model
- Look at the semantic similarity of the pooled outputs of different sentences

#### Note: This colab should be run with a GPU runtime

## Set up and imports

    pip install --quiet "tensorflow-text"

    import seaborn as sns
    from sklearn.metrics import pairwise

    import tensorflow as tf
    import tensorflow_hub as hub
    import tensorflow_text as text  # Imports TF ops for preprocessing.

```
/tmpfs/src/tf_docs_env/lib/python3.11/site-packages/tensorflow_hub/__init__.py:61: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  from pkg_resources import parse_version
```

### Configure the model

<br />

<button type="button" class="button-red button expand-control">Toggle code</button>

<br />

    BERT_MODEL = "https://tfhub.dev/google/experts/bert/wiki_books/2" # @param {type: "string"} ["https://tfhub.dev/google/experts/bert/wiki_books/2", "https://tfhub.dev/google/experts/bert/wiki_books/mnli/2", "https://tfhub.dev/google/experts/bert/wiki_books/qnli/2", "https://tfhub.dev/google/experts/bert/wiki_books/qqp/2", "https://tfhub.dev/google/experts/bert/wiki_books/squad2/2", "https://tfhub.dev/google/experts/bert/wiki_books/sst2/2",  "https://tfhub.dev/google/experts/bert/pubmed/2", "https://tfhub.dev/google/experts/bert/pubmed/squad2/2"]
    # Preprocessing must match the model, but all the above use the same.
    PREPROCESS_MODEL = "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3"

<br />

<br />

## Sentences

Let's take some sentences from Wikipedia to run through the model

    sentences = [
      "Here We Go Then, You And I is a 1999 album by Norwegian pop artist Morten Abel. It was Abel's second CD as a solo artist.",
      "The album went straight to number one on the Norwegian album chart, and sold to double platinum.",
      "Among the singles released from the album were the songs \"Be My Lover\" and \"Hard To Stay Awake\".",
      "Riccardo Zegna is an Italian jazz musician.",
      "Rajko Maksimović is a composer, writer, and music pedagogue.",
      "One of the most significant Serbian composers of our time, Maksimović has been and remains active in creating works for different ensembles.",
      "Ceylon spinach is a common name for several plants and may refer to: Basella alba Talinum fruticosum",
      "A solar eclipse occurs when the Moon passes between Earth and the Sun, thereby totally or partly obscuring the image of the Sun for a viewer on Earth.",
      "A partial solar eclipse occurs in the polar regions of the Earth when the center of the Moon's shadow misses the Earth.",
    ]

## Run the model

We'll load the BERT model from TF-Hub, tokenize our sentences using the matching preprocessing model from TF-Hub, then feed in the tokenized sentences to the model. To keep this colab fast and simple, we recommend running on GPU.

Go to **Runtime** → **Change runtime type** to make sure that **GPU** is selected

    preprocess = hub.load(PREPROCESS_MODEL)
    bert = hub.load(BERT_MODEL)
    inputs = preprocess(sentences)
    outputs = bert(inputs)

```
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1793708383.957783   25720 cuda_platform.cc:52] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
```

    print("Sentences:")
    print(sentences)

    print("\nBERT inputs:")
    print(inputs)

    print("\nPooled embeddings:")
    print(outputs["pooled_output"])

    print("\nPer token embeddings:")
    print(outputs["sequence_output"])

```
Sentences:
["Here We Go Then, You And I is a 1999 album by Norwegian pop artist Morten Abel. It was Abel's second CD as a solo artist.", 'The album went straight to number one on the Norwegian album chart, and sold to double platinum.', 'Among the singles released from the album were the songs "Be My Lover" and "Hard To Stay Awake".', 'Riccardo Zegna is an Italian jazz musician.', 'Rajko Maksimović is a composer, writer, and music pedagogue.', 'One of the most significant Serbian composers of our time, Maksimović has been and remains active in creating works for different ensembles.', 'Ceylon spinach is a common name for several plants and may refer to: Basella alba Talinum fruticosum', 'A solar eclipse occurs when the Moon passes between Earth and the Sun, thereby totally or partly obscuring the image of the Sun for a viewer on Earth.', "A partial solar eclipse occurs in the polar regions of the Earth when the center of the Moon's shadow misses the Earth."]

BERT inputs:
{'input_word_ids': <tf.Tensor: shape=(9, 128), dtype=int32, numpy=
array([[  101,  2182,  2057, ...,     0,     0,     0],
       [  101,  1996,  2201, ...,     0,     0,     0],
       [  101,  2426,  1996, ...,     0,     0,     0],
       ...,
       [  101, 16447,  6714, ...,     0,     0,     0],
       [  101,  1037,  5943, ...,     0,     0,     0],
       [  101,  1037,  7704, ...,     0,     0,     0]],
      shape=(9, 128), dtype=int32)>, 'input_type_ids': <tf.Tensor: shape=(9, 128), dtype=int32, numpy=
array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], shape=(9, 128), dtype=int32)>, 'input_mask': <tf.Tensor: shape=(9, 128), dtype=int32, numpy=
array([[1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       ...,
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0]], shape=(9, 128), dtype=int32)>}

Pooled embeddings:
tf.Tensor(
[[ 0.7975998  -0.48580435  0.4978166  ... -0.34488347  0.39727584
  -0.20639417]
 [ 0.57120407 -0.41205257  0.7048911  ... -0.35185158  0.19032468
  -0.40418985]
 [-0.6993832   0.15866981  0.06570031 ... -0.06232408 -0.81550235
  -0.07923666]
 ...
 [-0.35727316  0.77089757  0.1575639  ...  0.44185477 -0.86448294
   0.04504977]
 [ 0.9107701   0.4150139   0.560635   ... -0.49263898  0.3964051
  -0.05036213]
 [ 0.9050288  -0.15505227  0.72672135 ... -0.3473447   0.5052636
  -0.19542994]], shape=(9, 768), dtype=float32)

Per token embeddings:
tf.Tensor(
[[[ 1.09198034e+00 -5.30553818e-01  5.46399117e-01 ... -3.59624773e-01
    4.20410067e-01 -2.09402114e-01]
  [ 1.01438391e+00  7.80789614e-01  8.53757441e-01 ...  5.52821338e-01
   -1.12458062e+00  5.60275793e-01]
  [ 7.88626492e-01  7.77750984e-02  9.51508343e-01 ... -1.90754488e-01
    5.92059731e-01  6.19107544e-01]
  ...
  [-3.22028279e-01 -4.25213754e-01 -1.28239989e-01 ... -3.90947521e-01
   -7.90975094e-01  4.22367126e-01]
  [-3.10361311e-02  2.39854515e-01 -2.19943896e-01 ... -1.14401355e-01
   -1.26804888e+00 -1.61362067e-01]
  [-4.20634389e-01  5.49728811e-01 -3.24446052e-01 ... -1.84786558e-01
   -1.13429689e+00 -5.89751638e-02]]

 [[ 6.49308145e-01 -4.38081056e-01  8.76956046e-01 ... -3.67555380e-01
    1.92674041e-01 -4.28646863e-01]
  [-1.12487435e+00  2.99314171e-01  1.17996430e+00 ...  4.87294674e-01
    5.34003258e-01  2.28362232e-01]
  [-2.70571858e-01  3.23546529e-02  1.04256916e+00 ...  5.89937091e-01
    1.53678966e+00  5.84257245e-01]
  ...
  [-1.47624934e+00  1.82393149e-01  5.58769405e-02 ... -1.67331958e+00
   -6.73987389e-01 -7.24496961e-01]
  [-1.51381302e+00  5.81847191e-01  1.61419600e-01 ... -1.26408172e+00
   -4.02722955e-01 -9.71971750e-01]
  [-4.71529096e-01  2.28174269e-01  5.27763844e-01 ... -7.54835129e-01
   -9.09029603e-01 -1.69546157e-01]]

 [[-8.66092205e-01  1.60021871e-01  6.57950938e-02 ... -6.24049604e-02
   -1.14324021e+00 -7.94031247e-02]
  [ 7.71180272e-01  7.08046734e-01  1.13499373e-01 ...  7.88310945e-01
   -3.14381659e-01 -9.74873185e-01]
  [-4.40024555e-01 -3.00595462e-01  3.54795307e-01 ...  7.97378123e-02
   -4.73934829e-01 -1.10018504e+00]
  ...
  [-1.02052832e+00  2.69383162e-01 -4.73104864e-01 ... -6.63192987e-01
   -1.45799565e+00 -3.46653342e-01]
  [-9.70034778e-01 -4.50135916e-02 -5.97798705e-01 ... -3.05264354e-01
   -1.27442420e+00 -2.80516744e-01]
  [-7.31441498e-01  1.76993877e-01 -4.62579846e-01 ... -1.60622686e-01
   -1.63460875e+00 -3.20606112e-01]]

 ...

 [[-3.73756588e-01  1.02253628e+00  1.58887595e-01 ...  4.74533200e-01
   -1.31082082e+00  4.50802855e-02]
  [-4.15892422e-01  5.00192106e-01 -4.58442062e-01 ...  4.14824843e-01
   -6.20661736e-01 -7.15547979e-01]
  [-1.25044024e+00  5.09368300e-01 -5.71036935e-01 ...  3.54917735e-01
    2.43682057e-01 -2.05772042e+00]
  ...
  [ 1.33936316e-01  1.18591499e+00 -2.21702740e-01 ... -8.19468021e-01
   -1.67373490e+00 -3.96923363e-01]
  [-3.36623967e-01  1.65561700e+00 -3.78125399e-01 ... -9.67452526e-01
   -1.48010683e+00 -8.33310544e-01]
  [-2.26495236e-01  1.61784315e+00 -6.70443952e-01 ... -4.90784287e-01
   -1.45357466e+00 -7.17074811e-01]]

 [[ 1.53202260e+00  4.41653252e-01  6.33758664e-01 ... -5.39539039e-01
    4.19376582e-01 -5.04047722e-02]
  [ 8.93779159e-01  8.93955648e-01  3.06294113e-02 ...  5.90408146e-02
   -2.06495449e-01 -8.48112643e-01]
  [-1.85593031e-02  1.04790676e+00 -1.33295965e+00 ... -1.38696760e-01
   -3.78796816e-01 -4.90686804e-01]
  ...
  [ 1.42756486e+00  1.06969334e-01 -4.06334400e-02 ... -3.17781717e-02
   -4.14598584e-01  7.00371027e-01]
  [ 1.12866437e+00  1.45478234e-01 -6.13722205e-01 ...  4.74920094e-01
   -3.98519516e-01  4.31245565e-01]
  [ 1.43932879e+00  1.80306539e-01 -4.28541481e-01 ... -2.50227034e-01
   -1.00005531e+00  3.59855264e-01]]

 [[ 1.49933898e+00 -1.56313062e-01  9.21743810e-01 ... -3.62420946e-01
    5.56349099e-01 -1.97976559e-01]
  [ 1.11105216e+00  3.66514236e-01  3.55057210e-01 ... -5.42974174e-01
    1.44715160e-01 -3.16759408e-01]
  [ 2.40487456e-01  3.81157160e-01 -5.91829181e-01 ...  3.74108762e-01
   -5.98296762e-01 -1.01662636e+00]
  ...
  [ 1.01586187e+00  5.02603769e-01  1.07370466e-01 ... -9.56425607e-01
   -4.10394609e-01 -2.67601311e-01]
  [ 1.18489170e+00  6.54797554e-01  1.01521611e-03 ... -8.61546397e-01
   -8.80402923e-02 -3.06368530e-01]
  [ 1.26690960e+00  4.77678031e-01  6.62776828e-03 ... -1.15858161e+00
   -7.06755519e-02 -1.86787352e-01]]], shape=(9, 128, 768), dtype=float32)
```

## Semantic similarity

Now let's take a look at the `pooled_output` embeddings of our sentences and compare how similar they are across sentences.

### Helper functions

<br />

<button type="button" class="button-red button expand-control">Toggle code</button>

<br />

    def plot_similarity(features, labels):
      """Plot a similarity matrix of the embeddings."""
      cos_sim = pairwise.cosine_similarity(features)
      sns.set(font_scale=1.2)
      cbar_kws=dict(use_gridspec=False, location="left")
      g = sns.heatmap(
          cos_sim, xticklabels=labels, yticklabels=labels,
          vmin=0, vmax=1, cmap="Blues", cbar_kws=cbar_kws)
      g.tick_params(labelright=True, labelleft=False)
      g.set_yticklabels(labels, rotation=0)
      g.set_title("Semantic Textual Similarity")

<br />

<br />

    plot_similarity(outputs["pooled_output"], sentences)

![png](https://www.tensorflow.org/static/hub/tutorials/bert_experts_files/output_td6jcT0pJMZ5_0.png)

## Learn more

- Find more BERT models on [TensorFlow Hub](https://tfhub.dev)
- This notebook demonstrates simple inference with BERT, you can find a more advanced tutorial about fine-tuning BERT at [tensorflow.org/official_models/fine_tuning_bert](https://www.tensorflow.org/official_models/fine_tuning_bert)
- We used just one GPU chip to run the model, you can learn more about how to load models using tf.distribute at [tensorflow.org/tutorials/distribute/save_and_load](https://www.tensorflow.org/tutorials/distribute/save_and_load)