{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "headers"
},
"source": [
"Project: /text/_project.yaml\n",
"Book: /text/_book.yaml\n",
"\n",
" \n",
" \n",
"\n",
"\n",
"\n",
"\n",
"{% comment %}\n",
"The source of truth file can be found [here]: http://google3/third_party/tensorflow_text/g3doc\n",
"{% endcomment %}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "J0Qjg6vuaHNt"
},
"source": [
"# Neural machine translation with attention"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AOpGoE2T-YXS"
},
"source": [
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xh8WNEwYA3BW"
},
"source": [
"This tutorial demonstrates how to train a sequence-to-sequence (seq2seq) model for Spanish-to-English translation roughly based on [Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025v5) (Luong et al., 2015). \n",
"\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" \n",
"\n",
" This tutorial: An encoder/decoder connected by attention. \n",
" \n",
"
\n",
"\n",
"While this architecture is somewhat outdated, it is still a very useful project to work through to get a deeper understanding of sequence-to-sequence models and attention mechanisms (before going on to [Transformers](transformer.ipynb))."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CiwtNgENbx2g"
},
"source": [
"\n",
"\n",
"This example assumes some knowledge of TensorFlow fundamentals below the level of a Keras layer:\n",
" * [Working with tensors](https://www.tensorflow.org/guide/tensor) directly\n",
" * [Writing custom `keras.Model`s and `keras.layers`](https://www.tensorflow.org/guide/keras/custom_layers_and_models)\n",
"\n",
"After training the model in this notebook, you will be able to input a Spanish sentence, such as \"*¿todavia estan en casa?*\", and return the English translation: \"*are you still at home?*\"\n",
"\n",
"The resulting model is exportable as a tf.saved_model
, so it can be used in other TensorFlow environments.\n",
"\n",
"The translation quality is reasonable for a toy example, but the generated attention plot is perhaps more interesting. This shows which parts of the input sentence has the model's attention while translating:\n",
"\n",
" \n",
"\n",
"Note: This example takes approximately 10 minutes to run."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yAmSR1FaqKrl"
},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:11.074524Z",
"iopub.status.busy": "2023-12-07T12:22:11.073992Z",
"iopub.status.idle": "2023-12-07T12:22:15.707249Z",
"shell.execute_reply": "2023-12-07T12:22:15.706351Z"
},
"id": "DGFTkuRvzWqc"
},
"outputs": [],
"source": [
"!pip install \"tensorflow-text>=2.11\"\n",
"!pip install einops"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:15.711634Z",
"iopub.status.busy": "2023-12-07T12:22:15.711335Z",
"iopub.status.idle": "2023-12-07T12:22:18.532312Z",
"shell.execute_reply": "2023-12-07T12:22:18.531513Z"
},
"id": "tnxXKDjq3jEL"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:22:16.483003: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
"2023-12-07 12:22:16.483047: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
"2023-12-07 12:22:16.484590: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n"
]
}
],
"source": [
"import numpy as np\n",
"\n",
"import typing\n",
"from typing import Any, Tuple\n",
"\n",
"import einops\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.ticker as ticker\n",
"\n",
"import tensorflow as tf\n",
"import tensorflow_text as tf_text"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l_yq8kvIqoqQ"
},
"source": [
"This tutorial uses a lot of low level API's where it's easy to get shapes wrong. This class is used to check shapes throughout the tutorial."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:18.536971Z",
"iopub.status.busy": "2023-12-07T12:22:18.536253Z",
"iopub.status.idle": "2023-12-07T12:22:18.541973Z",
"shell.execute_reply": "2023-12-07T12:22:18.541286Z"
},
"id": "KqFqKi4fqN9X"
},
"outputs": [],
"source": [
"#@title\n",
"class ShapeChecker():\n",
" def __init__(self):\n",
" # Keep a cache of every axis-name seen\n",
" self.shapes = {}\n",
"\n",
" def __call__(self, tensor, names, broadcast=False):\n",
" if not tf.executing_eagerly():\n",
" return\n",
"\n",
" parsed = einops.parse_shape(tensor, names)\n",
"\n",
" for name, new_dim in parsed.items():\n",
" old_dim = self.shapes.get(name, None)\n",
" \n",
" if (broadcast and new_dim == 1):\n",
" continue\n",
"\n",
" if old_dim is None:\n",
" # If the axis name is new, add its length to the cache.\n",
" self.shapes[name] = new_dim\n",
" continue\n",
"\n",
" if new_dim != old_dim:\n",
" raise ValueError(f\"Shape mismatch for dimension: '{name}'\\n\"\n",
" f\" found: {new_dim}\\n\"\n",
" f\" expected: {old_dim}\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gjUROhJfH3ML"
},
"source": [
"## The data"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "puE_K74DIE9W"
},
"source": [
"The tutorial uses a language dataset provided by [Anki](http://www.manythings.org/anki/). This dataset contains language translation pairs in the format:\n",
"\n",
"```\n",
"May I borrow this book?\t¿Puedo tomar prestado este libro?\n",
"```\n",
"\n",
"They have a variety of languages available, but this example uses the English-Spanish dataset."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wfodePkj3jEa"
},
"source": [
"### Download and prepare the dataset\n",
"\n",
"For convenience, a copy of this dataset is hosted on Google Cloud, but you can also download your own copy. After downloading the dataset, here are the steps you need to take to prepare the data:\n",
"\n",
"1. Add a *start* and *end* token to each sentence.\n",
"2. Clean the sentences by removing special characters.\n",
"3. Create a word index and reverse word index (dictionaries mapping from word → id and id → word).\n",
"4. Pad each sentence to a maximum length."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:18.545276Z",
"iopub.status.busy": "2023-12-07T12:22:18.545023Z",
"iopub.status.idle": "2023-12-07T12:22:18.667063Z",
"shell.execute_reply": "2023-12-07T12:22:18.666394Z"
},
"id": "kRVATYOgJs1b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading data from http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2638744/2638744 [==============================] - 0s 0us/step\n"
]
}
],
"source": [
"# Download the file\n",
"import pathlib\n",
"\n",
"path_to_zip = tf.keras.utils.get_file(\n",
" 'spa-eng.zip', origin='http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip',\n",
" extract=True)\n",
"\n",
"path_to_file = pathlib.Path(path_to_zip).parent/'spa-eng/spa.txt'"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:18.670247Z",
"iopub.status.busy": "2023-12-07T12:22:18.669998Z",
"iopub.status.idle": "2023-12-07T12:22:18.674389Z",
"shell.execute_reply": "2023-12-07T12:22:18.673827Z"
},
"id": "OHn4Dct23jEm"
},
"outputs": [],
"source": [
"def load_data(path):\n",
" text = path.read_text(encoding='utf-8')\n",
"\n",
" lines = text.splitlines()\n",
" pairs = [line.split('\\t') for line in lines]\n",
"\n",
" context = np.array([context for target, context in pairs])\n",
" target = np.array([target for target, context in pairs])\n",
"\n",
" return target, context"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:18.677543Z",
"iopub.status.busy": "2023-12-07T12:22:18.676961Z",
"iopub.status.idle": "2023-12-07T12:22:19.206716Z",
"shell.execute_reply": "2023-12-07T12:22:19.205959Z"
},
"id": "cTbSbBz55QtF"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Si quieres sonar como un hablante nativo, debes estar dispuesto a practicar diciendo la misma frase una y otra vez de la misma manera en que un músico de banjo practica el mismo fraseo una y otra vez hasta que lo puedan tocar correctamente y en el tiempo esperado.\n"
]
}
],
"source": [
"target_raw, context_raw = load_data(path_to_file)\n",
"print(context_raw[-1])"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:19.209949Z",
"iopub.status.busy": "2023-12-07T12:22:19.209658Z",
"iopub.status.idle": "2023-12-07T12:22:19.213499Z",
"shell.execute_reply": "2023-12-07T12:22:19.212865Z"
},
"id": "lH_dPY8TRp3c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"If you want to sound like a native speaker, you must be willing to practice saying the same sentence over and over in the same way that banjo players practice the same phrase over and over until they can play it correctly and at the desired tempo.\n"
]
}
],
"source": [
"print(target_raw[-1])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rgCLkfv5uO3d"
},
"source": [
"### Create a tf.data dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PfVWx3WaI5Df"
},
"source": [
"From these arrays of strings you can create a tf.data.Dataset
of strings that shuffles and batches them efficiently:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:19.216791Z",
"iopub.status.busy": "2023-12-07T12:22:19.216230Z",
"iopub.status.idle": "2023-12-07T12:22:22.042643Z",
"shell.execute_reply": "2023-12-07T12:22:22.041764Z"
},
"id": "3rZFgz69nMPa"
},
"outputs": [],
"source": [
"BUFFER_SIZE = len(context_raw)\n",
"BATCH_SIZE = 64\n",
"\n",
"is_train = np.random.uniform(size=(len(target_raw),)) < 0.8\n",
"\n",
"train_raw = (\n",
" tf.data.Dataset\n",
" .from_tensor_slices((context_raw[is_train], target_raw[is_train]))\n",
" .shuffle(BUFFER_SIZE)\n",
" .batch(BATCH_SIZE))\n",
"val_raw = (\n",
" tf.data.Dataset\n",
" .from_tensor_slices((context_raw[~is_train], target_raw[~is_train]))\n",
" .shuffle(BUFFER_SIZE)\n",
" .batch(BATCH_SIZE))"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:22.046591Z",
"iopub.status.busy": "2023-12-07T12:22:22.046323Z",
"iopub.status.idle": "2023-12-07T12:22:22.178092Z",
"shell.execute_reply": "2023-12-07T12:22:22.177347Z"
},
"id": "qc6-NK1GtWQt"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tf.Tensor(\n",
"[b'\\xc2\\xbfQuieres ir de compras?' b'\\xc2\\xbfHay alguien en casa?'\n",
" b'Tom cre\\xc3\\xada que Mary lo tendr\\xc3\\xada dif\\xc3\\xadcil para conseguir una entrada para ese concierto.'\n",
" b'Mumbai es la capital del estado indio de Maharashtra.'\n",
" b'Tom me pidi\\xc3\\xb3 que le pasara la sal y la pimienta.'], shape=(5,), dtype=string)\n",
"\n",
"tf.Tensor(\n",
"[b'Do you want to go shopping?' b'Is anybody home?'\n",
" b'Tom thought it would be difficult for Mary to get a ticket to that concert.'\n",
" b'Mumbai is the capital of the Indian state of Maharashtra.'\n",
" b'Tom asked me to pass him the salt and pepper.'], shape=(5,), dtype=string)\n"
]
}
],
"source": [
"for example_context_strings, example_target_strings in train_raw.take(1):\n",
" print(example_context_strings[:5])\n",
" print()\n",
" print(example_target_strings[:5])\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zCoxLcuN3bwv"
},
"source": [
"### Text preprocessing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7kwdPcHvzz_a"
},
"source": [
"One of the goals of this tutorial is to build a model that can be exported as a tf.saved_model
. To make that exported model useful it should take tf.string
inputs, and return tf.string
outputs: All the text processing happens inside the model. Mainly using a layers.TextVectorization
layer."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EOQ5n55X4uDB"
},
"source": [
"#### Standardization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "upKhKAMK4zzI"
},
"source": [
"The model is dealing with multilingual text with a limited vocabulary. So it will be important to standardize the input text.\n",
"\n",
"The first step is Unicode normalization to split accented characters and replace compatibility characters with their ASCII equivalents.\n",
"\n",
"The `tensorflow_text` package contains a unicode normalize operation:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:22.182025Z",
"iopub.status.busy": "2023-12-07T12:22:22.181390Z",
"iopub.status.idle": "2023-12-07T12:22:22.186397Z",
"shell.execute_reply": "2023-12-07T12:22:22.185723Z"
},
"id": "mD0e-DWGQ2Vo"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"b'\\xc2\\xbfTodav\\xc3\\xada est\\xc3\\xa1 en casa?'\n",
"b'\\xc2\\xbfTodavi\\xcc\\x81a esta\\xcc\\x81 en casa?'\n"
]
}
],
"source": [
"example_text = tf.constant('¿Todavía está en casa?')\n",
"\n",
"print(example_text.numpy())\n",
"print(tf_text.normalize_utf8(example_text, 'NFKD').numpy())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6hTllEjK6RSo"
},
"source": [
"Unicode normalization will be the first step in the text standardization function:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:22.189478Z",
"iopub.status.busy": "2023-12-07T12:22:22.189225Z",
"iopub.status.idle": "2023-12-07T12:22:22.193927Z",
"shell.execute_reply": "2023-12-07T12:22:22.193252Z"
},
"id": "chTF5N885F0P"
},
"outputs": [],
"source": [
"def tf_lower_and_split_punct(text):\n",
" # Split accented characters.\n",
" text = tf_text.normalize_utf8(text, 'NFKD')\n",
" text = tf.strings.lower(text)\n",
" # Keep space, a to z, and select punctuation.\n",
" text = tf.strings.regex_replace(text, '[^ a-z.?!,¿]', '')\n",
" # Add spaces around punctuation.\n",
" text = tf.strings.regex_replace(text, '[.?!,¿]', r' \\0 ')\n",
" # Strip whitespace.\n",
" text = tf.strings.strip(text)\n",
"\n",
" text = tf.strings.join(['[START]', text, '[END]'], separator=' ')\n",
" return text"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:22.197138Z",
"iopub.status.busy": "2023-12-07T12:22:22.196577Z",
"iopub.status.idle": "2023-12-07T12:22:22.205704Z",
"shell.execute_reply": "2023-12-07T12:22:22.205030Z"
},
"id": "UREvDg3sEKYa"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"¿Todavía está en casa?\n",
"[START] ¿ todavia esta en casa ? [END]\n"
]
}
],
"source": [
"print(example_text.numpy().decode())\n",
"print(tf_lower_and_split_punct(example_text).numpy().decode())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4q-sKsSI7xRZ"
},
"source": [
"#### Text Vectorization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6aKn8qd37abi"
},
"source": [
"This standardization function will be wrapped up in a tf.keras.layers.TextVectorization
layer which will handle the vocabulary extraction and conversion of input text to sequences of tokens."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:22.208909Z",
"iopub.status.busy": "2023-12-07T12:22:22.208642Z",
"iopub.status.idle": "2023-12-07T12:22:22.222918Z",
"shell.execute_reply": "2023-12-07T12:22:22.222243Z"
},
"id": "eAY9k49G3jE_"
},
"outputs": [],
"source": [
"max_vocab_size = 5000\n",
"\n",
"context_text_processor = tf.keras.layers.TextVectorization(\n",
" standardize=tf_lower_and_split_punct,\n",
" max_tokens=max_vocab_size,\n",
" ragged=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7kbC6ODP8IK_"
},
"source": [
"The `TextVectorization` layer and many other [Keras preprocessing layers](https://www.tensorflow.org/guide/keras/preprocessing_layers) have an `adapt` method. This method reads one epoch of the training data, and works a lot like Model.fit
. This `adapt` method initializes the layer based on the data. Here it determines the vocabulary:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:22.226625Z",
"iopub.status.busy": "2023-12-07T12:22:22.225976Z",
"iopub.status.idle": "2023-12-07T12:22:24.601360Z",
"shell.execute_reply": "2023-12-07T12:22:24.600568Z"
},
"id": "bmsI1Yql8FYe"
},
"outputs": [
{
"data": {
"text/plain": [
"['', '[UNK]', '[START]', '[END]', '.', 'que', 'de', 'el', 'a', 'no']"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"context_text_processor.adapt(train_raw.map(lambda context, target: context))\n",
"\n",
"# Here are the first 10 words from the vocabulary:\n",
"context_text_processor.get_vocabulary()[:10]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9kGjIFjX8_Wp"
},
"source": [
"That's the Spanish `TextVectorization` layer, now build and `.adapt()` the English one:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:24.604989Z",
"iopub.status.busy": "2023-12-07T12:22:24.604641Z",
"iopub.status.idle": "2023-12-07T12:22:26.852853Z",
"shell.execute_reply": "2023-12-07T12:22:26.852115Z"
},
"id": "jlC4xuZnKLBS"
},
"outputs": [
{
"data": {
"text/plain": [
"['', '[UNK]', '[START]', '[END]', '.', 'the', 'i', 'to', 'you', 'tom']"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"target_text_processor = tf.keras.layers.TextVectorization(\n",
" standardize=tf_lower_and_split_punct,\n",
" max_tokens=max_vocab_size,\n",
" ragged=True)\n",
"\n",
"target_text_processor.adapt(train_raw.map(lambda context, target: target))\n",
"target_text_processor.get_vocabulary()[:10]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BWQqlP_s9eIv"
},
"source": [
"Now these layers can convert a batch of strings into a batch of token IDs:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:26.856522Z",
"iopub.status.busy": "2023-12-07T12:22:26.856004Z",
"iopub.status.idle": "2023-12-07T12:22:27.557144Z",
"shell.execute_reply": "2023-12-07T12:22:27.556492Z"
},
"id": "9KZxj8IrNZ9S"
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example_tokens = context_text_processor(example_context_strings)\n",
"example_tokens[:3, :]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AA9rUn9G9n78"
},
"source": [
"The `get_vocabulary` method can be used to convert token IDs back to text:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:27.560512Z",
"iopub.status.busy": "2023-12-07T12:22:27.560241Z",
"iopub.status.idle": "2023-12-07T12:22:27.577208Z",
"shell.execute_reply": "2023-12-07T12:22:27.576612Z"
},
"id": "98g9rcxGQY0I"
},
"outputs": [
{
"data": {
"text/plain": [
"'[START] ¿ quieres ir de compras ? [END]'"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"context_vocab = np.array(context_text_processor.get_vocabulary())\n",
"tokens = context_vocab[example_tokens[0].numpy()]\n",
"' '.join(tokens)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ot0aCL9t-Ghi"
},
"source": [
"The returned token IDs are zero-padded. This can easily be turned into a mask:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:27.580623Z",
"iopub.status.busy": "2023-12-07T12:22:27.580079Z",
"iopub.status.idle": "2023-12-07T12:22:27.907741Z",
"shell.execute_reply": "2023-12-07T12:22:27.906943Z"
},
"id": "_jx4Or_eFRSz"
},
"outputs": [
{
"data": {
"text/plain": [
"Text(0.5, 1.0, 'Mask')"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGzCAYAAACPa3XZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAyZUlEQVR4nO3de3xU1b3///dMQi6STCK3hAhEEAQvgBprjOIRITVERKigiNaiterxRFqIHnuoF8QfFa+AKJfqsfBAi1Y8R6gXoDQVPBaICOINRUAENCQoQkKAXGd//+DHmDEgrM1kzWTyej4e83iQPfOZtWZIPvlkzfrs7XEcxxEAAIAl3nBPAAAAtCwUHwAAwCqKDwAAYBXFBwAAsIriAwAAWEXxAQAArKL4AAAAVlF8AAAAqyg+AACAVRQfkCR5PB7deeed4Z4GABhZvny5PB6PXn311XBPBQYoPpoxj8dzXLfly5eHe6pG+vfvr7PPPjvo2Kmnnhp4PV6vV6mpqerdu7duu+02FRcXh2mmAObOnRv42Xz33Xcb3e84jjp37iyPx6Mrr7wyDDNEJIoN9wTg3gsvvBD09bx587Rs2bJGx8844wyb02oy55xzju666y5J0r59+/TZZ59pwYIFeu655zRu3DhNmTIlzDMEWq6EhATNnz9f/fr1Czq+YsUKff3114qPjw/TzBCJKD6asV/+8pdBX69evVrLli1rdDxanHLKKY1e26OPPqrrr79eU6dOVY8ePXTHHXeEaXZAy3bFFVdowYIFmj59umJjf/jVMn/+fGVlZem7774L4+wQafjYJcrt379fd911lzp37qz4+Hj17NlTTzzxhI7nYsaTJk2S1+vV008/HTi2ePFiXXLJJWrdurWSk5M1ePBgffrpp0FxN910k5KSkvTNN99o2LBhSkpKUvv27XX33Xervr4+pK8vMTFRL7zwgtq0aaM//vGPQa/r5ZdfVlZWlpKTk+Xz+dS7d2899dRTIR0fwCGjRo3S7t27tWzZssCxmpoavfrqq7r++usbPf6JJ57QRRddpLZt2yoxMVFZWVlH3LexbNky9evXT6mpqUpKSlLPnj31hz/84SfnUl1drSuvvFIpKSlauXLlib84hBzFRxRzHEdXXXWVpk6dqkGDBmnKlCnq2bOn/vM//1OFhYU/GXvffffpgQce0J/+9CeNGTNG0qGPeQYPHqykpCQ9+uijuv/++7Vhwwb169dPX331VVB8fX298vLy1LZtWz3xxBO69NJL9eSTT+rZZ58N+etMSkrSL37xC33zzTfasGGDpEMJa9SoUTr55JP16KOP6pFHHlH//v31r3/9K+TjAzi0LysnJ0cvvfRS4NjixYtVXl6u6667rtHjn3rqKZ177rl66KGH9PDDDys2NlbXXHON3nzzzcBjPv30U1155ZWqrq7WQw89pCeffFJXXXXVT/4cHzx4UEOGDNHKlSv1j3/8QxdddFFoXyhCw0HUKCgocBr+ly5cuNCR5EyaNCnocSNGjHA8Ho+zefPmwDFJTkFBgeM4jnPXXXc5Xq/XmTt3buD+ffv2Oampqc6tt94a9FylpaVOSkpK0PHRo0c7kpyHHnoo6LHnnnuuk5WVdczXcemllzpnnXVW0LHMzExn8ODBR42ZOnWqI8lZtGiR4ziO87vf/c7x+XxOXV3dMccD4N6cOXMcSc6aNWucZ555xklOTnYOHDjgOI7jXHPNNc5ll13mOE7jn+HDjzmspqbGOfvss50BAwYEjh3+uf7222+POv7bb7/tSHIWLFjg7Nu3z7n00kuddu3aOR988EEIXyVCjZWPKPbWW28pJiZGv/3tb4OO33XXXXIcR4sXLw467jiO7rzzTj311FN68cUXNXr06MB9y5Yt0969ezVq1Ch99913gVtMTIyys7P19ttvNxr/3//934O+vuSSS/Tll1+G8BX+ICkpSdKhjaiSlJqaqv379wctAQNoWtdee60OHjyoN954Q/v27dMbb7xxxI9cpEMfmR62Z88elZeX65JLLtG6desCx1NTUyVJixYtkt/v/8mxy8vLdfnll+vzzz/X8uXLdc4555zw60HTYcNpFNu2bZsyMjKUnJwcdPxw98u2bduCjs+bN0+VlZWaNWuWRo0aFXTfpk2bJEkDBgw44lg+ny/o64SEBLVv3z7o2Mknn6w9e/aYv5DjUFlZKUmB1/of//EfeuWVV5Sfn69TTjlFl19+ua699loNGjSoScYHILVv3165ubmaP3++Dhw4oPr6eo0YMeKIj33jjTc0adIkrV+/XtXV1YHjHo8n8O+RI0fqv//7v/Wb3/xG//Vf/6WBAwfq6quv1ogRI+T1Bv/tPHbsWFVVVemDDz7QWWed1TQvECHDygcCLr74YqWlpemZZ57R999/H3Tf4b86XnjhBS1btqzRbdGiRUGPj4mJsTZvSfrkk08kSd27d5ckdejQQevXr9ff/vY3XXXVVXr77beVn58ftJoDIPSuv/56LV68WLNnz1Z+fn5g9aKh//u//9NVV12lhIQEzZw5U2+99ZaWLVum66+/PmjTeGJiot555x394x//0I033qiPPvpII0eO1M9//vNGm9eHDh0qx3H0yCOPHHOVBOFH8RHFMjMzVVJSEvgo4rDPP/88cH9D3bt319///neVlJRo0KBBQXGnnXaapEO/1HNzcxvd+vfv37Qv5idUVlbqtddeU+fOnYPOaRIXF6chQ4Zo5syZ2rJli26//XbNmzdPmzdvDttcgWj3i1/8Ql6vV6tXrz7qRy7/8z//o4SEBC1dulS//vWvlZ+fr9zc3CM+1uv1auDAgZoyZYo2bNigP/7xj/rnP//Z6KPeYcOG6c9//rPmz5+vgoKCkL8uhBbFRxS74oorVF9fr2eeeSbo+NSpU+XxeJSfn98opk+fPnrrrbf02WefaciQITp48KAkKS8vTz6fTw8//LBqa2sbxX377bdN8yKO4eDBg7rxxhv1/fff69577w0s2e7evTvocV6vV3369JGkoCVeAKGVlJSkWbNm6cEHH9SQIUOO+JiYmBh5PJ6g1YuvvvpKCxcuDHrcj1dgJQX2chzp5/hXv/qVpk+frtmzZ+v3v/+9+xeBJseejyg2ZMgQXXbZZbr33nv11VdfqW/fvvr73/+uRYsWaezYsYHVjB+78MILtWjRIl1xxRUaMWKEFi5cKJ/Pp1mzZunGG2/Ueeedp+uuu07t27fX9u3b9eabb+riiy9uVOSE2jfffKMXX3xR0qHVjg0bNmjBggUqLS3VXXfdpdtvvz3w2N/85jf6/vvvNWDAAHXq1Enbtm3T008/rXPOOSdqzvgKRKpjfbw5ePBgTZkyRYMGDdL111+vXbt2acaMGerevbs++uijwOMeeughvfPOOxo8eLAyMzO1a9cuzZw5U506dWp0JtXD7rzzTlVUVOjee+9VSkrKMc8JgjAJb7MNQunHrbaOc6hFdty4cU5GRobTqlUrp0ePHs7jjz/u+P3+oMepQavtYYsWLXJiY2OdkSNHOvX19Y7jHGpry8vLc1JSUpyEhATntNNOc2666Sbn/fffD8SNHj3aad26daP5TZgwodH8juRorbaSHEmOx+NxfD6fc9ZZZzm33nqrU1xc3Og5Xn31Vefyyy93OnTo4MTFxTldunRxbr/9dmfnzp3HHB/A8WvYavtTftxq+/zzzzs9evRw4uPjnV69ejlz5sxplCOKioqcoUOHOhkZGU5cXJyTkZHhjBo1yvniiy8Cj2nYatvQPffc40hynnnmmRC9UoSSx3GO41SXAAAAIcKeDwAAYBXFBwAAsIriAwAAWEXxAQAArKL4AAAAVlF8AAAAqyLuJGN+v18lJSVKTk4OusAQAHscx9G+ffuUkZHR6AJekYrcAYSXSd6IuOKjpKREnTt3Dvc0AEjasWOHOnXqFO5pHBdyBxAZjidvRFzxcfiS6P10hWLVqsnHi800T1Z123YYx3jj441jJMnv4jok3rg483FqaoxjEL3qVKt39Vbg57E5ODzXbetOlS+peazWIDL94vTe4Z5Cs2SSNyKu+Di8XBqrVor1WCg+vC6KAhfz8nrMCwJJ8nvMLw3tdTE/v4cT3aKB///boTl9fHF4rr4kr3zJMWGeDZozG797opJB3uDPAwAAYBXFBwAAsCriPnaxre6rbVbG8bROdBdYXeViMGpKANElL6NvuKeAEOK3FAAAsIriAwAAWEXxAQAArKL4AAAAVlF8AAAAq6Kq2yW2x2nGMZ+NP9k4psev3zeOUU2teYxLdTlnGsd4l68zj0l018HT613z92JDVp2rsQBEh6UlH4Z7Cj+JbhwzrHwAAACrKD4AAIBVFB8AAMAqig8AAGAVxQcAALAqqrpdVFNjHHLGf241jnHTd+E5yeW1XSorjUPcdK644T940FXchqwQTwRA2NDlATdY+QAAAFYZFx/ffPONfvnLX6pt27ZKTExU79699f77P5z3wnEcPfDAA+rYsaMSExOVm5urTZs2hXTSAJofcgeAw4yKjz179ujiiy9Wq1attHjxYm3YsEFPPvmkTj75hxN1PfbYY5o+fbpmz56t4uJitW7dWnl5eaqqcnFpeABRgdwBoCGjPR+PPvqoOnfurDlz5gSOde3aNfBvx3E0bdo03XfffRo6dKgkad68eUpLS9PChQt13XXXhWjaAJoTcgeAhoxWPv72t7/p/PPP1zXXXKMOHTro3HPP1XPPPRe4f+vWrSotLVVubm7gWEpKirKzs7Vq1aojPmd1dbUqKiqCbgCiC7kDQENGKx9ffvmlZs2apcLCQv3hD3/QmjVr9Nvf/lZxcXEaPXq0SktLJUlpaWlBcWlpaYH7fmzy5MmaOHGiy+kHq9u2wzjGExNjHBOTmmoc89mjmcYxktTj5m9dxQGRJNJzBwC7jFY+/H6/zjvvPD388MM699xzddttt+nWW2/V7NmzXU9g/PjxKi8vD9x27DAvIABENnIHgIaMio+OHTvqzDODr5h6xhlnaPv27ZKk9PR0SVJZWVnQY8rKygL3/Vh8fLx8Pl/QDUB0IXcAaMio+Lj44ou1cePGoGNffPGFMjMPfaTQtWtXpaenq6ioKHB/RUWFiouLlZOTE4LpAmiOyB0AGjLa8zFu3DhddNFFevjhh3Xttdfqvffe07PPPqtnn31WkuTxeDR27FhNmjRJPXr0UNeuXXX//fcrIyNDw4YNa4r5A2gGyB0AGjIqPn72s5/ptdde0/jx4/XQQw+pa9eumjZtmm644YbAY+655x7t379ft912m/bu3at+/fppyZIlSkhICPnkATQP5A4ADXkcx3HCPYmGKioqlJKSov4aqlhPK7Ngj/nZ4j1ej3GMU19vHOOWNy7OOMbv4ho3QEN1Tq2Wa5HKy8ubzV6Kw7ljzxfd5Es272KDO1zbBYeZ5A2u7QIAAKyi+AAAAFZRfAAAAKsoPgAAgFVG3S4Rz/Gbx5zfxzym+CPzGJc8LjacysWGUzenmfcmJhrHSFL9/gPGMZG+MRhoqZaWfBjuKTRbLXmzLisfAADAKooPAABgFcUHAACwiuIDAABYRfEBAACsiq5uFxe+GpxkHJNZbD7O97+5yDxIUpv/Xmkc46ZzxU1nyDnv7jOOkaR155t38Dh1ta7GAgAbWnLnihusfAAAAKsoPgAAgFUUHwAAwCqKDwAAYBXFBwAAsKrFd7t0e3WPcYynY7pxTLu/fGAcI0kurlZj7Zoma88xv97KIXSuANGCLg+4wcoHAACwiuIDAABYRfEBAACsovgAAABWUXwAAACrWny3i77cYRzir60zj6muMo6JdJ7YVq7iuE4LED2WlnwY7imEHB08TY+VDwAAYBXFBwAAsIriAwAAWEXxAQAArIqqDacxycnGMZ609sYx/q3bjGOiERtHgejBJkvYxMoHAACwiuIDAABYRfEBAACsovgAAABWUXwAAACroqrbZedNvY1jOjyz2nwgx28eY1FMUpJxTH1lZRPMBEBzEY2nSXeLzp+mx8oHAACwiuIDAABYRfEBAACsovgAAABWUXwAAACroqrbpcPTK8M9haP69j8uchXXfqb5a6JzBUAko5sErHwAAACrKD4AAIBVFB8AAMAqig8AAGAVxQcAALAqqrpd5DGvpaqu+plxzElLzK+BkLa63DhGkvwuXpOba894zje/Lo7z/sfGMQBg8zoydNZEJlY+AACAVUbFx4MPPiiPxxN069WrV+D+qqoqFRQUqG3btkpKStLw4cNVVlYW8kkDaF7IHQAaMl75OOuss7Rz587A7d133w3cN27cOL3++utasGCBVqxYoZKSEl199dUhnTCA5oncAeAw4z0fsbGxSk9Pb3S8vLxczz//vObPn68BAwZIkubMmaMzzjhDq1ev1oUXXnjiswXQbJE7ABxmvPKxadMmZWRkqFu3brrhhhu0fft2SdLatWtVW1ur3NzcwGN79eqlLl26aNWqVUd9vurqalVUVATdAEQfcgeAw4xWPrKzszV37lz17NlTO3fu1MSJE3XJJZfok08+UWlpqeLi4pSamhoUk5aWptLS0qM+5+TJkzVx4kRXkw+FvTfvM47Zkd/HOKbnLJfXW3HRueJqGDpX0ISiMXfAPTpQYFR85OfnB/7dp08fZWdnKzMzU6+88ooSExNdTWD8+PEqLCwMfF1RUaHOnTu7ei4AkYncAaChE2q1TU1N1emnn67NmzcrPT1dNTU12rt3b9BjysrKjvg572Hx8fHy+XxBNwDRjdwBtGwnVHxUVlZqy5Yt6tixo7KystSqVSsVFRUF7t+4caO2b9+unJycE54ogOhB7gBaNqOPXe6++24NGTJEmZmZKikp0YQJExQTE6NRo0YpJSVFt9xyiwoLC9WmTRv5fD6NGTNGOTk57FYHWjhyB4CGjIqPr7/+WqNGjdLu3bvVvn179evXT6tXr1b79u0lSVOnTpXX69Xw4cNVXV2tvLw8zZw5s0kmHiqdbt1lHFP//UbjGL/fMY6RJG9cnHHM9nvON47pNGmlcQxwvKIxd8A9N6dXZ5NqdPE4juPut2ITqaioUEpKivprqGI9rcyCXVwHJbZdG+OY+u/3GMc4bouPVuaX36H4wImqc2q1XItUXl7ebPZSHM4de77oJl9yTLingxCj+Ih8JnmDa7sAAACrKD4AAIBVFB8AAMAqig8AAGCV+W7GCObxeoxj3vyw6NgP+hE3G59iu51qHCNJdV9+ZRzD5lEAtrARFG6w8gEAAKyi+AAAAFZRfAAAAKsoPgAAgFUUHwAAwKqo6nZxw81O7ZjkZOOY+m07jGNcc3GaeTn+0M8DQNRzc50Wm+jGiUysfAAAAKsoPgAAgFUUHwAAwCqKDwAAYBXFBwAAsCqqul1iOqabB9XXG4fU7Sw1H8cmOleAFo0OD0Q6Vj4AAIBVFB8AAMAqig8AAGAVxQcAALAqqjac+r/fYx5z4EATzKQxT0yMqzjHxYZYb2KicYz/4EHjGFencZcUm97BOKaudJf5QC423nrj4szHkeSvqXEVBzSFSD/lOeyLtE3IrHwAAACrKD4AAIBVFB8AAMAqig8AAGAVxQcAALAqqrpdVG/e3RCb5qLzYtd3xjHf/TrbOEaS2j630jjGTefKnptzjGNOnrPKOEaK7NPT07UCRJdI6/LAIax8AAAAqyg+AACAVRQfAADAKooPAABgFcUHAACwKqq6XfzVVcYxzne1xjEer8c4pt2cNcYxkuS4uH6KNzHBOMZt5woA2EDXSnRh5QMAAFhF8QEAAKyi+AAAAFZRfAAAAKsoPgAAgFVR1e3ihpPd2zxo1UfGIbGnZ5qPI6lu42bjGP+BA67GAgBTdKHADVY+AACAVRQfAADAKooPAABgFcUHAACwiuIDAABYFVXdLp7YVsYxMaV7jWPqjCOkrSM7uIiSOj9k3u0CAEAkO6GVj0ceeUQej0djx44NHKuqqlJBQYHatm2rpKQkDR8+XGVlZSc6TwBRgrwBwHXxsWbNGv3pT39Snz59go6PGzdOr7/+uhYsWKAVK1aopKREV1999QlPFEDzR94AILksPiorK3XDDTfoueee08knnxw4Xl5erueff15TpkzRgAEDlJWVpTlz5mjlypVavXp1yCYNoPkhbwA4zFXxUVBQoMGDBys3Nzfo+Nq1a1VbWxt0vFevXurSpYtWrVp1xOeqrq5WRUVF0A1A9All3pDIHUBzZrzh9OWXX9a6deu0Zs2aRveVlpYqLi5OqampQcfT0tJUWlp6xOebPHmyJk6caDqNI6pd0tE4xsn9ynwgj3nNlvGvavNxJMVmdjaOqdu2w9VYpvaPuNBVXL97zf+a/eBc83EqfpljHON78ei/7OBeqPOGFNrcAfeWlnwY7ingOETaafCNfovu2LFDv/vd7/SXv/xFCQkJIZnA+PHjVV5eHrjt2GHnFycAO5oib0jkDqA5Myo+1q5dq127dum8885TbGysYmNjtWLFCk2fPl2xsbFKS0tTTU2N9u7dGxRXVlam9PT0Iz5nfHy8fD5f0A1A9GiKvCGRO4DmzOhjl4EDB+rjjz8OOnbzzTerV69e+v3vf6/OnTurVatWKioq0vDhwyVJGzdu1Pbt25WTY74EDqD5I28A+DGj4iM5OVlnn3120LHWrVurbdu2geO33HKLCgsL1aZNG/l8Po0ZM0Y5OTm68EJ3+wMANG/kDQA/FvIznE6dOlVer1fDhw9XdXW18vLyNHPmzFAPAyCKkDeAlsXjOI4T7kk0VFFRoZSUFPXXUMV6zE+XHqlizu7pKq7+003GMc7FfY79oB/x/Osj4xg5fvMYSd7EROMY/8GDrsaCO3VOrZZrkcrLy5vNXorDuWPPF93kS44J93SAY4q0DpQTZZI3uLAcAACwiuIDAABYRfEBAACsovgAAABWUXwAAACrQt5qG1YurrkS276tcUzdrm+NY+o/2WgcI0nek04yjvG/u944JiYlxTimvrzcOEaS3tyy0jgmP8PFxV1cODgs21Vc4sLiEM8EQLSzeV2cSOusYeUDAABYRfEBAACsovgAAABWUXwAAACrKD4AAIBVUdXt4m1l/nLcdK7seLW3ccypt31tHCNJnhjza1T4DxwwjnHbueKGrc4VN+haAZpepHVewD5WPgAAgFUUHwAAwCqKDwAAYBXFBwAAsCqqNpz6a2qMY7zxCcYxnUd8bBxTbxwBAJGPzaNwg5UPAABgFcUHAACwiuIDAABYRfEBAACsovgAAABWRVW3iysxkV1/xaZ1MI6pK9vVBDMBgMaWlnxoHEOHDCL7Ny8AAIg6FB8AAMAqig8AAGAVxQcAALCK4gMAAFjV4rtd/AerjGNi27czjqn/fo9xjGSvcyW2Z3fjmLqNm5tgJgCinZsOGbforIlMrHwAAACrKD4AAIBVFB8AAMAqig8AAGAVxQcAALCqxXe7KLu3cUjdavOd2rvGXGQcI0kdnl7pKs4UnSsA3KCbBG6w8gEAAKyi+AAAAFZRfAAAAKsoPgAAgFUtfsOp96NNxjF+F+MkfVPvIkqKSU01jqnt2804xrtinXEMANg6VTobW6MLKx8AAMAqig8AAGAVxQcAALCK4gMAAFhF8QEAAKyKqm4XT2wr4xj/gQNNMJPGkt74wFVcfU2NcQydKwBsoQsFbrDyAQAArDIqPmbNmqU+ffrI5/PJ5/MpJydHixcvDtxfVVWlgoICtW3bVklJSRo+fLjKyspCPmkAzQu5A0BDRsVHp06d9Mgjj2jt2rV6//33NWDAAA0dOlSffvqpJGncuHF6/fXXtWDBAq1YsUIlJSW6+uqrm2TiAJoPcgeAhjyO4zgn8gRt2rTR448/rhEjRqh9+/aaP3++RowYIUn6/PPPdcYZZ2jVqlW68MILj+v5KioqlJKSov4aqliP2R4ON3s+nLpa4xg3vHFxruL8LvZ8ACeqzqnVci1SeXm5fD5fk4zRVLljzxfd5EuOaZI5ozH2fOAwk7zhes9HfX29Xn75Ze3fv185OTlau3atamtrlZubG3hMr1691KVLF61ateqoz1NdXa2KioqgG4DoRe4AYNzt8vHHHysnJ0dVVVVKSkrSa6+9pjPPPFPr169XXFycUn90LZK0tDSVlpYe9fkmT56siRMnGk/8iBzzq664uXbK59O6G8f0uOl94xjJ3WpOTIr5X6p13+8xH6f1ScYxkuT4zf+fbHUloelEdO6Aa7au7RLpWAEyY7zy0bNnT61fv17FxcW64447NHr0aG3YsMH1BMaPH6/y8vLAbceOHa6fC0DkIncAOMx45SMuLk7dux/6yz8rK0tr1qzRU089pZEjR6qmpkZ79+4N+gumrKxM6enpR32++Ph4xcfHm88cQLNC7gBw2Amf58Pv96u6ulpZWVlq1aqVioqKAvdt3LhR27dvV05OzokOAyDKkDuAlsto5WP8+PHKz89Xly5dtG/fPs2fP1/Lly/X0qVLlZKSoltuuUWFhYVq06aNfD6fxowZo5ycnOPerQ4gOpE7ADRkVHzs2rVLv/rVr7Rz506lpKSoT58+Wrp0qX7+859LkqZOnSqv16vhw4erurpaeXl5mjlzZpNMHEDzQe4A0NAJn+cj1E7kPB+xPU4zHq9u0xbjGCDa2TjPR6hxno8TQ7cGTpSV83wAAAC4QfEBAACsovgAAABWUXwAAACrKD4AAIBVxmc4jWgV+8xjPC7qLzfXkElJMR9HUn15uas4ADDh9hotdMnADVY+AACAVRQfAADAKooPAABgFcUHAACwKqo2nNaV7bIyTmyvHsYxdZ9vcjXWpud/Zhxz0iaz09JL0imPrDSOAQC3G1VNsbE1urDyAQAArKL4AAAAVlF8AAAAqyg+AACAVRQfAADAqqjqdnGj4oYc4xjfX1aZD+TmNO6SfO0qjWM63rHFOMb8hPHuxWZ2No6Z8X8vGcfc3qWfcQyAyGSrq8amltzBw8oHAACwiuIDAABYRfEBAACsovgAAABWUXwAAACrWny3y8nrdxvHfD7nfOOYrn8xDpEknfKbr4xj/C47a0x54+JcxdV/XWIcQ+cKgEjWkjtX3GDlAwAAWEXxAQAArKL4AAAAVlF8AAAAqyg+AACAVS2+28Wz63vjmC7/k2ocE/uP94xjJKnOVZQd/pqacE8BQIjQrQGbWPkAAABWUXwAAACrKD4AAIBVFB8AAMCqFr/htO7b74xjxkw13zz67JvdjWMkSY7fXZwhN6dKd7vhtHTcRcYxHacXG8c49fXGMbGnZBjHSFLdN+anjAciydKSD8M9BTShSNtQzMoHAACwiuIDAABYRfEBAACsovgAAABWUXwAAACroqrbxVbHxrM9uhnHeBPjjWMkyX/woKs443FcvA9uO0PSp602jonplmkcU/Os+cnp6wbuMI4BAJsirXPFDVY+AACAVRQfAADAKooPAABgFcUHAACwiuIDAABYFV3dLl27GMf4N242jvHExJiPY6lrxa3tE82vt9JlwsommMmR1W3ZahzjHdgEEwHQrERDZ0g0YuUDAABYZVR8TJ48WT/72c+UnJysDh06aNiwYdq4cWPQY6qqqlRQUKC2bdsqKSlJw4cPV1lZWUgnDaB5IXcAaMio+FixYoUKCgq0evVqLVu2TLW1tbr88su1f//+wGPGjRun119/XQsWLNCKFStUUlKiq6++OuQTB9B8kDsANGS052PJkiVBX8+dO1cdOnTQ2rVr9W//9m8qLy/X888/r/nz52vAgAGSpDlz5uiMM87Q6tWrdeGFF4Zu5gCaDXIHgIZOaM9HeXm5JKlNmzaSpLVr16q2tla5ubmBx/Tq1UtdunTRqlWrjvgc1dXVqqioCLoBiG7kDqBlc93t4vf7NXbsWF188cU6++yzJUmlpaWKi4tTampq0GPT0tJUWlp6xOeZPHmyJk6c6HYaQZz4ViF5nmOO43eMY2I7neJqrLqvvzEP8pjXlN1e+tY4xvzKKUBk5g4Adrle+SgoKNAnn3yil19++YQmMH78eJWXlwduO3ZwYS8gmpE7ALha+bjzzjv1xhtv6J133lGnTp0Cx9PT01VTU6O9e/cG/QVTVlam9PT0Iz5XfHy84uPdXfEVQPNC7gAgGa58OI6jO++8U6+99pr++c9/qmvXrkH3Z2VlqVWrVioqKgoc27hxo7Zv366cnJzQzBhAs0PuANCQ0cpHQUGB5s+fr0WLFik5OTnwWWxKSooSExOVkpKiW265RYWFhWrTpo18Pp/GjBmjnJwcdqsDLRi5A0BDRsXHrFmzJEn9+/cPOj5nzhzddNNNkqSpU6fK6/Vq+PDhqq6uVl5enmbOnBmSyQJonsgdABryOI5j3rrRhCoqKpSSkqL+GqpYj1n3yrb/z/z6JO0/qDeOOel/i41jgOakzqnVci1SeXm5fD5fuKdzXA7njj1fdJMv2fz6S9GE65kgHEzyBtd2AQAAVlF8AAAAqyg+AACAVRQfAADAKtenV49EmQ+sNg9y/MYh+643P+9A8vwjX58CAEJtacmHxjFsUoVNrHwAAACrKD4AAIBVFB8AAMAqig8AAGAVxQcAALAqqrpd3HSuxJ6SYRzjqnPF467O2/JktnHMaYXm84tpcBnz41W/d69xDIDI5KZDpjmgiycysfIBAACsovgAAABWUXwAAACrKD4AAIBVFB8AAMCq6Op2ccH/7XdWxtk/4gJXcW46V9ygcwWAG3STwA1WPgAAgFUUHwAAwCqKDwAAYBXFBwAAsIriAwAAWEW3S22dlXGS/neNqzjHRYwnu4950NrPjENiUnzm40iq273bVRyAyBON14Shg6fpsfIBAACsovgAAABWUXwAAACrKD4AAIBVUbXh1BsXZxzjr6kxjolJSjKOcU7PNI6RJGfdp+YxxR+5GssUG0cBsDkTbrDyAQAArKL4AAAAVlF8AAAAqyg+AACAVRQfAADAqqjqdnHTuSKPef1VndPLOCZ22fvGMZIU272bcUzdlq+MY+r7n2scE/P2WuMYANEl0k+vTjdOZGLlAwAAWEXxAQAArKL4AAAAVlF8AAAAqyg+AACAVVHV7eKK4zcOcdO54omJMY6RpLrNX7qKMxX7znrjGMftYC46jNz8PwGArW4cumrMsPIBAACsovgAAABWUXwAAACrKD4AAIBVFB8AAMCq6Op2cdFFEdP6JOOY+v0HjGO8ycnGMZIkx7ynpL683HyY+nrjGNfoXAFaNDpDwMoHAACwyrj4eOeddzRkyBBlZGTI4/Fo4cKFQfc7jqMHHnhAHTt2VGJionJzc7Vp06ZQzRdAM0TeANCQcfGxf/9+9e3bVzNmzDji/Y899pimT5+u2bNnq7i4WK1bt1ZeXp6qqqpOeLIAmifyBoCGjPd85OfnKz8//4j3OY6jadOm6b777tPQoUMlSfPmzVNaWpoWLlyo66677sRmC6BZIm8AaCikez62bt2q0tJS5ebmBo6lpKQoOztbq1atOmJMdXW1Kioqgm4AWg43eUMidwDNWUi7XUpLSyVJaWlpQcfT0tIC9/3Y5MmTNXHixNBMwEUXRe2ik41jYi4/aBxTv3evcYxNu357kXFMh+krXY1VNTTbOCZhUbGrsRD53OQNKcS5A1bZut6KTXTwmAl7t8v48eNVXl4euO3YsSPcUwLQDJA7gOYrpMVHenq6JKmsrCzoeFlZWeC+H4uPj5fP5wu6AWg53OQNidwBNGchLT66du2q9PR0FRUVBY5VVFSouLhYOTk5oRwKQJQgbwAtj/Gej8rKSm3evDnw9datW7V+/Xq1adNGXbp00dixYzVp0iT16NFDXbt21f3336+MjAwNGzYslPMG0IyQNwA0ZFx8vP/++7rssssCXxcWFkqSRo8erblz5+qee+7R/v37ddttt2nv3r3q16+flixZooSEhNDNOoTifm0e8+VfzzSO6TLiY/OBLHK7edQNNo+2PNGWN/ADNlrCDY/juLh4SBOqqKhQSkqK+muoYj2tmny82MzOxjFfPplqHBPpxQfQUJ1Tq+VapPLy8mazl+Jw7tjzRTf5kmPCPZ0Wg+IDh5nkjbB3uwAAgJaF4gMAAFhF8QEAAKyi+AAAAFaF9PTq4eaNizOOqdtmflbELiPMY2KSkoxjJKm+stJVHAAAkYqVDwAAYBXFBwAAsIriAwAAWEXxAQAArKL4AAAAVkVVt4u/ts44Jrbj0S/ZfTR1O0uNY9x2rcRmdDSOqSvZ6WosazzmNW+ML9k4pr683DgGgJmlJR+GewrNVks+NT0rHwAAwCqKDwAAYBXFBwAAsIriAwAAWEXxAQAArIqqbhc5fvOQAwebYCKNubnujCRVXNjFOOak/43wbhcX/090rgDRoyV3eeAQVj4AAIBVFB8AAMAqig8AAGAVxQcAALAqujacuuBmI2NsWgfjmLqyXcYxknTSa2uMY6qHXGAcE//6e8Yxnuw+xjGS5Lz3iYsg802qm2ZkG8f0KCg2jgFghlOyn5ho2LDLygcAALCK4gMAAFhF8QEAAKyi+AAAAFZRfAAAAKtafLeLG246V8pvzHE1Vup88y6UxCLzbhLzXhLJKf7IRZQ9dK4AsCUaOlBsYuUDAABYRfEBAACsovgAAABWUXwAAACrKD4AAIBVLb7bpeoq8+ugJPzNvAMl5UV3nReOi5jNfzTfdX36gxuMY+or9hnHSJLngrONY9xcD6biBvNru/j+4u7/ydvKxY/S2T2MQ/zrPjUfB0CT43o1UsW+ep18+vE9lpUPAABgFcUHAACwiuIDAABYRfEBAACsovgAAABWtfhul8S3PjCO8SYlGcfUV1Yax0iSJybGOOa0cauMY+qNI9yzdU0Y34vm74Nb/poa8yA6V9BCcR2U6FTn1Er68rgey8oHAACwiuIDAABYRfEBAACsovgAAABWUXwAAACrWny3i8fFNTn8VdVNMJMjq/75ecYxcUvWNMFMAAAIjSZb+ZgxY4ZOPfVUJSQkKDs7W++9Z34xNgAtC3kDaBmapPj461//qsLCQk2YMEHr1q1T3759lZeXp127djXFcACiAHkDaDmapPiYMmWKbr31Vt18880688wzNXv2bJ100kn685//3BTDAYgC5A2g5Qj5no+amhqtXbtW48ePDxzzer3Kzc3VqlWNzzhZXV2t6uof9lCUl5dLkupUKzmhnl1jXsf8DKKOY34+UMepNY6RpLraKuMYr8uxgMPqdOh7yHEs/BDKPG9IR88dFZX+pp0sTlgdOSoqmeSNkBcf3333nerr65WWlhZ0PC0tTZ9//nmjx0+ePFkTJ05sdPxdvRXqqR3ZQTvDuPaPReGeAVqwffv2KSUlpcnHMc0b0tFzR+Z5XzXFFBFSx3cKbjRPx5M3wt7tMn78eBUWFga+3rt3rzIzM7V9+3YrSS+SVVRUqHPnztqxY4d8Pl+4pxM2vA+H2HwfHMfRvn37lJGR0aTjnAhyx5Hx8/ID3otDbL0PJnkj5MVHu3btFBMTo7KysqDjZWVlSk9Pb/T4+Ph4xcfHNzqekpLSor9ZGvL5fLwX4n04zNb7YPMXuGnekMgdx8LPyw94Lw6x8T4cb94I+YbTuLg4ZWVlqaioKHDM7/erqKhIOTk5oR4OQBQgbwAtS5N87FJYWKjRo0fr/PPP1wUXXKBp06Zp//79uvnmm5tiOABRgLwBtBxNUnyMHDlS3377rR544AGVlpbqnHPO0ZIlSxptJjuS+Ph4TZgw4YjLqS0N78UhvA+HRPv7cCJ5Q4r+9+d48T78gPfikEh8HzyOrV46AAAAcWE5AABgGcUHAACwiuIDAABYRfEBAACsovgAAABWRVzxMWPGDJ166qlKSEhQdna23nvvvXBPyaoHH3xQHo8n6NarV69wT8uKd955R0OGDFFGRoY8Ho8WLlwYdL/jOHrggQfUsWNHJSYmKjc3V5s2bQrPZJvQsd6Hm266qdH3yKBBg8Iz2QjR0vOG1HJzB3njB80pd0RU8fHXv/5VhYWFmjBhgtatW6e+ffsqLy9Pu3btCvfUrDrrrLO0c+fOwO3dd98N95Ss2L9/v/r27asZM2Yc8f7HHntM06dP1+zZs1VcXKzWrVsrLy9PVVXmV/6NZMd6HyRp0KBBQd8jL730ksUZRhbyxg9aYu4gb/ygWeUOJ4JccMEFTkFBQeDr+vp6JyMjw5k8eXIYZ2XXhAkTnL59+4Z7GmEnyXnttdcCX/v9fic9Pd15/PHHA8f27t3rxMfHOy+99FIYZmjHj98Hx3Gc0aNHO0OHDg3LfCIReeMQcgd5o6FIzx0Rs/JRU1OjtWvXKjc3N3DM6/UqNzdXq1atCuPM7Nu0aZMyMjLUrVs33XDDDdq+fXu4pxR2W7duVWlpadD3R0pKirKzs1vc94ckLV++XB06dFDPnj11xx13aPfu3eGeUliQN4KRO4KRNxqLlNwRMcXHd999p/r6+kanUk5LS1NpaWmYZmVfdna25s6dqyVLlmjWrFnaunWrLrnkEu3bty/cUwurw98DLf37Qzq0bDpv3jwVFRXp0Ucf1YoVK5Sfn6/6+vpwT8068sYPyB2NkTeCRVLuaJJru8C9/Pz8wL/79Omj7OxsZWZm6pVXXtEtt9wSxpkhUlx33XWBf/fu3Vt9+vTRaaedpuXLl2vgwIFhnBnCidyBY4mk3BExKx/t2rVTTEyMysrKgo6XlZUpPT09TLMKv9TUVJ1++unavHlzuKcSVoe/B/j+aKxbt25q165di/weIW8cHbmDvHEs4cwdEVN8xMXFKSsrS0VFRYFjfr9fRUVFysnJCePMwquyslJbtmxRx44dwz2VsOratavS09ODvj8qKipUXFzcor8/JOnrr7/W7t27W+T3CHnj6Mgd5I1jCWfuiKiPXQoLCzV69Gidf/75uuCCCzRt2jTt379fN998c7inZs3dd9+tIUOGKDMzUyUlJZowYYJiYmI0atSocE+tyVVWVgZV4Fu3btX69evVpk0bdenSRWPHjtWkSZPUo0cPde3aVffff78yMjI0bNiw8E26CfzU+9CmTRtNnDhRw4cPV3p6urZs2aJ77rlH3bt3V15eXhhnHT7kjUNaau4gb/ygWeWOcLfb/NjTTz/tdOnSxYmLi3MuuOACZ/Xq1eGeklUjR450Onbs6MTFxTmnnHKKM3LkSGfz5s3hnpYVb7/9tiOp0W306NGO4xxqm7v//vudtLQ0Jz4+3hk4cKCzcePG8E66CfzU+3DgwAHn8ssvd9q3b++0atXKyczMdG699VantLQ03NMOq5aeNxyn5eYO8sYPmlPu8DiO49gtdwAAQEsWMXs+AABAy0DxAQAArKL4AAAAVlF8AAAAqyg+AACAVRQfAADAKooPAABgFcUHAACwiuIDAABYRfEBAACsovgAAABW/T/dVhjXUKS+zAAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.subplot(1, 2, 1)\n",
"plt.pcolormesh(example_tokens.to_tensor())\n",
"plt.title('Token IDs')\n",
"\n",
"plt.subplot(1, 2, 2)\n",
"plt.pcolormesh(example_tokens.to_tensor() != 0)\n",
"plt.title('Mask')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3O0B4XdFlRgc"
},
"source": [
"### Process the dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rVCuyuSp_whd"
},
"source": [
"The `process_text` function below converts the `Datasets` of strings, into 0-padded tensors of token IDs. It also converts from a `(context, target)` pair to an `((context, target_in), target_out)` pair for training with keras.Model.fit
. Keras expects `(inputs, labels)` pairs, the inputs are the `(context, target_in)` and the labels are `target_out`. The difference between `target_in` and `target_out` is that they are shifted by one step relative to eachother, so that at each location the label is the next token."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:27.911404Z",
"iopub.status.busy": "2023-12-07T12:22:27.911139Z",
"iopub.status.idle": "2023-12-07T12:22:28.115341Z",
"shell.execute_reply": "2023-12-07T12:22:28.114646Z"
},
"id": "wk5tbZWQl5u1"
},
"outputs": [],
"source": [
"def process_text(context, target):\n",
" context = context_text_processor(context).to_tensor()\n",
" target = target_text_processor(target)\n",
" targ_in = target[:,:-1].to_tensor()\n",
" targ_out = target[:,1:].to_tensor()\n",
" return (context, targ_in), targ_out\n",
"\n",
"\n",
"train_ds = train_raw.map(process_text, tf.data.AUTOTUNE)\n",
"val_ds = val_raw.map(process_text, tf.data.AUTOTUNE)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4iGi7X2m_tbM"
},
"source": [
"Here is the first sequence of each, from the first batch:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:28.119272Z",
"iopub.status.busy": "2023-12-07T12:22:28.118779Z",
"iopub.status.idle": "2023-12-07T12:22:28.313215Z",
"shell.execute_reply": "2023-12-07T12:22:28.312401Z"
},
"id": "woQBWAjLsJkr"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ 2 32 28 365 16 434 8 11 505 6]\n",
"\n",
"[ 2 32 215 10 505 25 47 218 580 4]\n",
"[ 32 215 10 505 25 47 218 580 4 3]\n"
]
}
],
"source": [
"for (ex_context_tok, ex_tar_in), ex_tar_out in train_ds.take(1):\n",
" print(ex_context_tok[0, :10].numpy()) \n",
" print()\n",
" print(ex_tar_in[0, :10].numpy()) \n",
" print(ex_tar_out[0, :10].numpy()) "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TNfHIF71ulLu"
},
"source": [
"## The encoder/decoder\n",
"\n",
"The following diagrams shows an overview of the model. In both the encoder is on the left, the decoder is on the right. At each time-step the decoder's output is combined with the encoder's output, to predict the next word. \n",
"\n",
"The original [left] contains a few extra connections that are intentionally omitted from this tutorial's model [right], as they are generally unnecessary, and difficult to implement. Those missing connections are:\n",
"\n",
"1. Feeding the state from the encoder's RNN to the decoder's RNN\n",
"2. Feeding the attention output back to the RNN's input.\n",
"\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
"\n",
" The original from Effective Approaches to Attention-based Neural Machine Translation \n",
" This tutorial's model \n",
" \n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gzQWx2saImMV"
},
"source": [
"Before getting into it define constants for the model:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:28.317005Z",
"iopub.status.busy": "2023-12-07T12:22:28.316322Z",
"iopub.status.idle": "2023-12-07T12:22:28.319746Z",
"shell.execute_reply": "2023-12-07T12:22:28.319160Z"
},
"id": "_a9uNz3-IrF-"
},
"outputs": [],
"source": [
"UNITS = 256"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "blNgVbLSzpsr"
},
"source": [
"### The encoder\n",
"\n",
"The goal of the encoder is to process the context sequence into a sequence of vectors that are useful for the decoder as it attempts to predict the next output for each timestep. Since the context sequence is constant, there is no restriction on how information can flow in the encoder, so use a bidirectional-RNN to do the processing:\n",
"\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" \n",
"\n",
" A bidirectional RNN \n",
" \n",
"
\n",
"\n",
"The encoder:\n",
"\n",
"1. Takes a list of token IDs (from `context_text_processor`).\n",
"3. Looks up an embedding vector for each token (Using a layers.Embedding
).\n",
"4. Processes the embeddings into a new sequence (Using a bidirectional layers.GRU
).\n",
"5. Returns the processed sequence. This will be passed to the attention head."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:28.323345Z",
"iopub.status.busy": "2023-12-07T12:22:28.322724Z",
"iopub.status.idle": "2023-12-07T12:22:28.329737Z",
"shell.execute_reply": "2023-12-07T12:22:28.329065Z"
},
"id": "nZ2rI24i3jFg"
},
"outputs": [],
"source": [
"class Encoder(tf.keras.layers.Layer):\n",
" def __init__(self, text_processor, units):\n",
" super(Encoder, self).__init__()\n",
" self.text_processor = text_processor\n",
" self.vocab_size = text_processor.vocabulary_size()\n",
" self.units = units\n",
" \n",
" # The embedding layer converts tokens to vectors\n",
" self.embedding = tf.keras.layers.Embedding(self.vocab_size, units,\n",
" mask_zero=True)\n",
"\n",
" # The RNN layer processes those vectors sequentially.\n",
" self.rnn = tf.keras.layers.Bidirectional(\n",
" merge_mode='sum',\n",
" layer=tf.keras.layers.GRU(units,\n",
" # Return the sequence and state\n",
" return_sequences=True,\n",
" recurrent_initializer='glorot_uniform'))\n",
"\n",
" def call(self, x):\n",
" shape_checker = ShapeChecker()\n",
" shape_checker(x, 'batch s')\n",
"\n",
" # 2. The embedding layer looks up the embedding vector for each token.\n",
" x = self.embedding(x)\n",
" shape_checker(x, 'batch s units')\n",
"\n",
" # 3. The GRU processes the sequence of embeddings.\n",
" x = self.rnn(x)\n",
" shape_checker(x, 'batch s units')\n",
"\n",
" # 4. Returns the new sequence of embeddings.\n",
" return x\n",
"\n",
" def convert_input(self, texts):\n",
" texts = tf.convert_to_tensor(texts)\n",
" if len(texts.shape) == 0:\n",
" texts = tf.convert_to_tensor(texts)[tf.newaxis]\n",
" context = self.text_processor(texts).to_tensor()\n",
" context = self(context)\n",
" return context"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D3SKkaQeGn-Q"
},
"source": [
"Try it out:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:28.333093Z",
"iopub.status.busy": "2023-12-07T12:22:28.332575Z",
"iopub.status.idle": "2023-12-07T12:22:29.046111Z",
"shell.execute_reply": "2023-12-07T12:22:29.045369Z"
},
"id": "60gSVh05Jl6l"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Context tokens, shape (batch, s): (64, 18)\n",
"Encoder output, shape (batch, s, units): (64, 18, 256)\n"
]
}
],
"source": [
"# Encode the input sequence.\n",
"encoder = Encoder(context_text_processor, UNITS)\n",
"ex_context = encoder(ex_context_tok)\n",
"\n",
"print(f'Context tokens, shape (batch, s): {ex_context_tok.shape}')\n",
"print(f'Encoder output, shape (batch, s, units): {ex_context.shape}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "45xM_Gl1MgXY"
},
"source": [
"### The attention layer\n",
"\n",
"The attention layer lets the decoder access the information extracted by the encoder. It computes a vector from the entire context sequence, and adds that to the decoder's output. \n",
"\n",
"The simplest way you could calculate a single vector from the entire sequence would be to take the average across the sequence (layers.GlobalAveragePooling1D
). An attention layer is similar, but calculates a **weighted** average across the context sequence. Where the weights are calculated from the combination of context and \"query\" vectors.\n",
"\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" \n",
"\n",
" The attention layer \n",
" \n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.050088Z",
"iopub.status.busy": "2023-12-07T12:22:29.049552Z",
"iopub.status.idle": "2023-12-07T12:22:29.055725Z",
"shell.execute_reply": "2023-12-07T12:22:29.055145Z"
},
"id": "-Ql3ymqwD8LS"
},
"outputs": [],
"source": [
"class CrossAttention(tf.keras.layers.Layer):\n",
" def __init__(self, units, **kwargs):\n",
" super().__init__()\n",
" self.mha = tf.keras.layers.MultiHeadAttention(key_dim=units, num_heads=1, **kwargs)\n",
" self.layernorm = tf.keras.layers.LayerNormalization()\n",
" self.add = tf.keras.layers.Add()\n",
"\n",
" def call(self, x, context):\n",
" shape_checker = ShapeChecker()\n",
" \n",
" shape_checker(x, 'batch t units')\n",
" shape_checker(context, 'batch s units')\n",
"\n",
" attn_output, attn_scores = self.mha(\n",
" query=x,\n",
" value=context,\n",
" return_attention_scores=True)\n",
" \n",
" shape_checker(x, 'batch t units')\n",
" shape_checker(attn_scores, 'batch heads t s')\n",
" \n",
" # Cache the attention scores for plotting later.\n",
" attn_scores = tf.reduce_mean(attn_scores, axis=1)\n",
" shape_checker(attn_scores, 'batch t s')\n",
" self.last_attention_weights = attn_scores\n",
"\n",
" x = self.add([x, attn_output])\n",
" x = self.layernorm(x)\n",
"\n",
" return x"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.058971Z",
"iopub.status.busy": "2023-12-07T12:22:29.058489Z",
"iopub.status.idle": "2023-12-07T12:22:29.356131Z",
"shell.execute_reply": "2023-12-07T12:22:29.355469Z"
},
"id": "7y7hjPkNMmHh"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Context sequence, shape (batch, s, units): (64, 18, 256)\n",
"Target sequence, shape (batch, t, units): (64, 16, 256)\n",
"Attention result, shape (batch, t, units): (64, 16, 256)\n",
"Attention weights, shape (batch, t, s): (64, 16, 18)\n"
]
}
],
"source": [
"attention_layer = CrossAttention(UNITS)\n",
"\n",
"# Attend to the encoded tokens\n",
"embed = tf.keras.layers.Embedding(target_text_processor.vocabulary_size(),\n",
" output_dim=UNITS, mask_zero=True)\n",
"ex_tar_embed = embed(ex_tar_in)\n",
"\n",
"result = attention_layer(ex_tar_embed, ex_context)\n",
"\n",
"print(f'Context sequence, shape (batch, s, units): {ex_context.shape}')\n",
"print(f'Target sequence, shape (batch, t, units): {ex_tar_embed.shape}')\n",
"print(f'Attention result, shape (batch, t, units): {result.shape}')\n",
"print(f'Attention weights, shape (batch, t, s): {attention_layer.last_attention_weights.shape}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vx9fUhi3Pmwp"
},
"source": [
"The attention weights will sum to `1` over the context sequence, at each location in the target sequence."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.359942Z",
"iopub.status.busy": "2023-12-07T12:22:29.359404Z",
"iopub.status.idle": "2023-12-07T12:22:29.365050Z",
"shell.execute_reply": "2023-12-07T12:22:29.364488Z"
},
"id": "zxyR7cmQPn9P"
},
"outputs": [
{
"data": {
"text/plain": [
"array([1. , 0.9999999 , 1. , 1. , 0.99999994,\n",
" 0.99999994, 1. , 1. , 1. , 1. ,\n",
" 1. , 1. , 1. , 1. , 1. ,\n",
" 1. ], dtype=float32)"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"attention_layer.last_attention_weights[0].numpy().sum(axis=-1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AagyXMH-Jhqt"
},
"source": [
"\n",
"\n",
"Here are the attention weights across the context sequences at `t=0`:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.368783Z",
"iopub.status.busy": "2023-12-07T12:22:29.368214Z",
"iopub.status.idle": "2023-12-07T12:22:29.629324Z",
"shell.execute_reply": "2023-12-07T12:22:29.628459Z"
},
"id": "Rqr8XGsAJlf6"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGzCAYAAACPa3XZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAzGklEQVR4nO3deXRUdZ7//1elyGY2CISEDCSsGtTGJWoIAo2QJjKIICCLzjR4aHWcSDdEx56oiHFQ1FahsVnUn4PHaRBFWdQZoTWyqIRFVNrWBkFBaEMC0pCELQnJ5/eHX6pTJCgVKp9blXo+zqlzqHtv1X3fCrx55VP3c6/LGGMEAABgSZjTBQAAgNBC+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfiAJMnlcunhhx92ugy/mzhxojp37tzk18bGxvq3IAB+tXbtWrlcLr3++utOlwIfED78YN68eXK5XMrKymp0/ZdffqmHH35Ye/bsafS1L730UvMW+P/83//9X4sMGE47fvy4Hn74Ya1du9bpUgDrXnrpJblcLrlcLn344YcN1htj1KlTJ7lcLt1www0OVIhARPjwg0WLFqlz587avHmzdu3a1WD9l19+qcLCwoAIH4WFhY2uO3HihB588EErddj0wgsvaMeOHc26j+PHj6uwsJDwgZAWFRWlxYsXN1i+bt06/e1vf1NkZKQDVSFQET7O0+7du7VhwwY988wzSkpK0qJFi5wuqUmioqLUqlUrp8vwu/DwcJoeYME///M/a+nSpTp16pTX8sWLFyszM1MpKSkOVYZARPg4T4sWLVKbNm00dOhQjR49ukH4eOmll3TzzTdLkq677jrP8OTatWvVuXNnffHFF1q3bp1n+YABAzyvPXLkiKZMmaJOnTopMjJS3bt31xNPPKG6ujrPNnv27JHL5dJTTz2l559/Xt26dVNkZKSuvvpqbdmyxbPdxIkTNXfuXEny7MvlcnnWN3bOx6effqohQ4YoPj5esbGxGjRokDZu3Njg+Fwulz766CPl5+crKSlJMTExuummm3Tw4MEf/ezefPNNuVwu/fnPf/Yse+ONN+RyuTRy5EivbXv27KmxY8d6LfvjH/+ozMxMRUdHKzExUePGjdO+ffu8tmnsnI9Dhw7pX//1XxUfH6/WrVtrwoQJ2rZtm1wuV6OjUN99951GjBih2NhYJSUl6d5771Vtba2kHz7/pKQkSVJhYaHncz39WZaWluq2225Tx44dFRkZqQ4dOmj48OGNjoIBwWz8+PE6dOiQ3n33Xc+y6upqvf7667rlllsabP/UU0+pT58+atu2raKjo5WZmdnoeRvvvvuu+vbtq9atWys2NlYXXXSR7r///h+tpaqqSjfccIMSEhK0YcOG8z84+F3L+1XXskWLFmnkyJGKiIjQ+PHjNX/+fG3ZskVXX321JKl///769a9/rTlz5uj+++9Xz549Jf3wn+ns2bM1efJkxcbG6oEHHpAkJScnS/phKP/nP/+5vvvuO915551KS0vThg0bVFBQoP3792v27NledSxevFiVlZW688475XK59OSTT2rkyJH65ptvFB4erjvvvFMlJSV699139T//8z8/eVxffPGF+vXrp/j4eN13330KDw/Xc889pwEDBmjdunUNzm+ZPHmy2rRpo+nTp2vPnj2aPXu27r77br366qtn3Uffvn3lcrm0fv169erVS5L0wQcfKCwszOu744MHD2r79u26++67PcseffRRTZs2TWPGjNGvfvUrHTx4UM8++6z69++vTz/9VK1bt250n3V1dRo2bJg2b96su+66SxkZGVq5cqUmTJjQ6Pa1tbXKzc1VVlaWnnrqKb333nt6+umn1a1bN911111KSkrS/Pnzddddd+mmm27yhKbTxzNq1Ch98cUXmjx5sjp37qwDBw7o3Xff1d69e5t8IiwQiDp37qzs7Gy98sorGjJkiCTpnXfeUXl5ucaNG6c5c+Z4bf/73/9eN954o2699VZVV1dryZIluvnmm/X2229r6NChkn7oQzfccIN69eqlRx55RJGRkdq1a5c++uijs9Zx4sQJDR8+XB9//LHee+89Ty9GgDFoso8//thIMu+++64xxpi6ujrTsWNH85vf/MZru6VLlxpJZs2aNQ3e45JLLjE///nPGyz/r//6LxMTE2O++uorr+X/+Z//adxut9m7d68xxpjdu3cbSaZt27bm73//u2e7lStXGknmrbfe8izLy8szZ/uRSzLTp0/3PB8xYoSJiIgwX3/9tWdZSUmJiYuLM/379/csW7hwoZFkcnJyTF1dnWf51KlTjdvtNkeOHGl0f/WPf8yYMZ7nV155pbn55puNJPPXv/7VGGPMsmXLjCSzbds2Y4wxe/bsMW632zz66KNe7/X555+bVq1aeS2fMGGCSU9P9zx/4403jCQze/Zsz7La2lozcOBAI8ksXLjQ67WSzCOPPOK1nyuuuMJkZmZ6nh88eLDB52eMMYcPHzaSzO9+97sf/QyAYHa6B2zZssX84Q9/MHFxceb48ePGGGNuvvlmc9111xljjElPTzdDhw71vO70NqdVV1ebSy+91AwcONCzbNasWUaSOXjw4Fn3v2bNGiPJLF261FRWVpqf//znpl27dubTTz/141HC3/ja5TwsWrRIycnJuu666yT98NXF2LFjtWTJEs+wfFMtXbpU/fr1U5s2bfT99997Hjk5OaqtrdX69eu9th87dqzatGnjed6vXz9J0jfffOPzvmtra/WnP/1JI0aMUNeuXT3LO3TooFtuuUUffvihKioqvF5zxx13eH2N069fP9XW1urbb7/90X3169dPH3zwgSSpsrJS27Zt0x133KF27dp5ln/wwQdq3bq1Lr30UknSsmXLVFdXpzFjxnh9NikpKerRo4fWrFlz1v2tWrVK4eHhuv322z3LwsLClJeXd9bX/Nu//VuDms/lc42OjlZERITWrl2rw4cP/+T2QLAbM2aMTpw4obfffluVlZV6++23G/3KRfrh38dphw8fVnl5ufr166dPPvnEs/z0CObKlSu9vm5uTHl5uQYPHqzt27dr7dq1uvzyy8/7eNB8CB9NVFtbqyVLlui6667T7t27tWvXLu3atUtZWVkqKytTUVHReb3/zp07tWrVKiUlJXk9cnJyJEkHDhzw2j4tLc3r+ekg0pT/9A4ePKjjx4/roosuarCuZ8+eqqura3BuRVP3369fP+3fv1+7du3Shg0b5HK5lJ2d7RVKPvjgA1177bUKC/vhr+vOnTtljFGPHj0afD5//etfG3w29X377bfq0KGDLrjgAq/l3bt3b3T7qKgozzkd9Y/tXD7XyMhIPfHEE3rnnXeUnJys/v3768knn1RpaelPvhYIRqd71OLFi7Vs2TLV1tZq9OjRjW779ttvq3fv3oqKilJiYqLnK8zy8nLPNmPHjtW1116rX/3qV0pOTta4ceP02muvNRpEpkyZoi1btui9997TJZdc0mzHCP/gnI8mev/997V//34tWbJES5YsabB+0aJFGjx4cJPfv66uTr/4xS903333Nbr+wgsv9Hrudrsb3c4Y0+QafNHU/fft21eStH79en3zzTe68sorFRMTo379+mnOnDk6evSoPv30Uz366KOe19TV1cnlcumdd95pdL/+vDDY2Y7rXE2ZMkXDhg3TihUrtHr1ak2bNk0zZ87U+++/ryuuuMJPVQKB45ZbbtHtt9+u0tJSDRkypNHzrz744APdeOON6t+/v+bNm6cOHTooPDxcCxcu9JquGx0drfXr12vNmjX63//9X61atUqvvvqqBg4cqD/96U9e/z6HDx+uJUuW6PHHH9fLL7/s+WUFgYnw0USLFi1S+/btPTNI6lu2bJmWL1+uBQsWKDo62uvriDOdbV23bt109OhRz0iHP/xYHfUlJSXpggsuaPT6GNu3b1dYWJg6derkl5rS0tKUlpamDz74QN98843n66L+/fsrPz9fS5cuVW1trfr37+95Tbdu3WSMUZcuXRqEsJ+Snp6uNWvW6Pjx416jH41dn+Vc/dTn2q1bN91zzz265557tHPnTl1++eV6+umn9cc//rHJ+wQC1U033aQ777xTGzduPOsJ52+88YaioqK0evVqr6nwCxcubLBtWFiYBg0apEGDBumZZ57RY489pgceeEBr1qzx6o8jRozQ4MGDNXHiRMXFxWn+/Pn+Pzj4DdGwCU6cOKFly5bphhtu0OjRoxs87r77blVWVurNN9+UJMXExEj6YersmWJiYhpdPmbMGBUXF2v16tUN1h05cqTBXPpz8WN11Od2uzV48GCtXLnSa0poWVmZFi9erL59+yo+Pt7n/Z9Nv3799P7772vz5s2e8HH55ZcrLi5Ojz/+uGca3mkjR46U2+1WYWFhg5EVY4wOHTp01n3l5uaqpqZGL7zwgmdZXV1doyHyXJ0OMWd+rsePH9fJkye9lnXr1k1xcXGqqqpq8v6AQBYbG6v58+fr4Ycf1rBhwxrdxu12y+VyeZ0bt2fPHq1YscJru7///e8NXnv6XI7G/g398pe/1Jw5c7RgwQL99re/bfpBoNkx8tEEb775piorK3XjjTc2ur53796eC46NHTtWl19+udxut5544gmVl5crMjJSAwcOVPv27ZWZman58+drxowZ6t69u9q3b6+BAwfqP/7jP/Tmm2/qhhtu0MSJE5WZmaljx47p888/1+uvv649e/aoXbt2PtV9+j/wX//618rNzZXb7da4ceMa3XbGjBme+fX//u//rlatWum5555TVVWVnnzySd8+sJ/Qr18/LVq0SC6Xy/M1jNvtVp8+fbR69WoNGDBAERERnu27deumGTNmqKCgQHv27NGIESMUFxen3bt3a/ny5brjjjt07733NrqvESNG6JprrtE999yjXbt2KSMjQ2+++aanyZ3r6FB90dHRuvjii/Xqq6/qwgsvVGJioi699FKdOnVKgwYN0pgxY3TxxRerVatWWr58ucrKys76uQMtwdmmrp82dOhQPfPMM7r++ut1yy236MCBA5o7d666d+/udd2fRx55ROvXr9fQoUOVnp6uAwcOaN68eerYsaOnV5zp7rvvVkVFhR544AElJCT85DVB4BAnp9oEq2HDhpmoqChz7Nixs24zceJEEx4ebr7//ntjjDEvvPCC6dq1q3G73V7TbktLS83QoUNNXFyckeQ17baystIUFBSY7t27m4iICNOuXTvTp08f89RTT5nq6mpjzD+m2jY2nVNnTP88deqUmTx5sklKSjIul8tr2u2Z2xpjzCeffGJyc3NNbGysueCCC8x1111nNmzY4LVN/Wl29Z2e/tbY9OIzffHFF0aS6dmzp9fyGTNmGElm2rRpjb7ujTfeMH379jUxMTEmJibGZGRkmLy8PLNjxw7PNmdOtTXmh6mxt9xyi4mLizMJCQlm4sSJ5qOPPjKSzJIlS7xeGxMT02C/06dPbzBlecOGDSYzM9NERER4Psvvv//e5OXlmYyMDBMTE2MSEhJMVlaWee21137yMwGCxdl6wJnOnGr74osvmh49epjIyEiTkZFhFi5c2ODfVlFRkRk+fLhJTU01ERERJjU11YwfP97rEgT1p9rWd9999xlJ5g9/+IOfjhT+5DLG0hmJQABbsWKFbrrpJn344Ye69tprnS4HAFo0wgdCzokTJ7yuMVBbW6vBgwfr448/Vmlpqdc6AID/cc4HQs7kyZN14sQJZWdnq6qqSsuWLdOGDRv02GOPETwAwAJGPhByFi9erKefflq7du3SyZMn1b17d911111e944BADQfwgcAALCK63wAAACrCB8AAMCqgDvhtK6uTiUlJYqLi2vSBZ8AnD9jjCorK5Wamho098igdwDO8qVvBFz4KCkp8dt9QwCcn3379qljx45Ol3FO6B1AYDiXvhFw4SMuLk6S9O0nnRUfGxy/cfnTTRf+zOkSAJ1SjT7U/3n+PQYDege9A87ypW8EXPg4PVwaHxum+Ljzu515MGrlCne6BED6f3PggunrC3oHvQMO86FvhN6vBwAAwFGEDwAAYFXAfe1y2uXLJiksKsrpMuyb5XQBZ+Fq/mvRdZuysdn3AbRUq0u2OV2CI3JTL3O6BDQBIx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwKqAne1yYa+9Co+JcLqMFql6wH6nSwAQhJhZAn9h5AMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWBWws12++nNaaN7bxYZZnZ2uoFHdphY7XQKAHxGI949hBk5wYuQDAABY5XP4+O677/Qv//Ivatu2raKjo/Wzn/1MH3/8sWe9MUYPPfSQOnTooOjoaOXk5Gjnzp1+LRpA8KF3ADjNp/Bx+PBhXXvttQoPD9c777yjL7/8Uk8//bTatGnj2ebJJ5/UnDlztGDBAm3atEkxMTHKzc3VyZMn/V48gOBA7wBQn0/nfDzxxBPq1KmTFi5c6FnWpUsXz5+NMZo9e7YefPBBDR8+XJL08ssvKzk5WStWrNC4ceP8VDaAYELvAFCfTyMfb775pq666irdfPPNat++va644gq98MILnvW7d+9WaWmpcnJyPMsSEhKUlZWl4uLGTyasqqpSRUWF1wNAy0LvAFCfTyMf33zzjebPn6/8/Hzdf//92rJli379618rIiJCEyZMUGlpqSQpOTnZ63XJycmedWeaOXOmCgsLGyyP3e2SO9LlS3kIcgcm93G6BL9o/+wGp0sIODZ7B0JLIM7AaYpQm7Xj08hHXV2drrzySj322GO64oordMcdd+j222/XggULmlxAQUGBysvLPY99+/Y1+b0ABCZ6B4D6fAofHTp00MUXX+y1rGfPntq7d68kKSUlRZJUVlbmtU1ZWZln3ZkiIyMVHx/v9QDQstA7ANTnU/i49tprtWPHDq9lX331ldLT0yX9cAJZSkqKioqKPOsrKiq0adMmZWdn+6FcAMGI3gGgPp/O+Zg6dar69Omjxx57TGPGjNHmzZv1/PPP6/nnn5ckuVwuTZkyRTNmzFCPHj3UpUsXTZs2TampqRoxYkRz1A8gCNA7ANTnU/i4+uqrtXz5chUUFOiRRx5Rly5dNHv2bN16662ebe677z4dO3ZMd9xxh44cOaK+fftq1apViuJS6UDIoncAqM9ljDFOF1FfRUWFEhISlDH5MbkjaTpwVoenQ3PmyilTo7VaqfLy8qA5l+J07zj8VVfFx7mdLgchLtRmr0i+9Q3u7QIAAKwifAAAAKsIHwAAwCrCBwAAsMqn2S42dRryrcJjIpwuo0WqHrDf6RIABKFQPIkSzYORDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgVcDOdtl9KFHuE+d+eXWXy7erxBvj8rWkluP1dj6/pNPoz5uhEADBZHXJNp+2Z3YMzoaRDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgVcDOdqmqiFJYzbnPdkHz2vni1T5t32PSlmaqBECw8HV2jMQMmVDByAcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsCpgZ7tkdClReEyE02W0SNUD9jtdAoAgxEwU+AsjHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwL2hNOvi9PljuLy6s1iRhenK/CL9Ac3OF0CEFKacrn0QMSJs85j5AMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWBWws11qY4xMlHG6DPtcPh6zcTVPHQ7oNrXY6RIABBlmrgQnRj4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFUBO9vls5EvKj7O7XQZ1nHmNoCmoHcgmDDyAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsCtjZLldvGiv3BVFOl+HF5eN9V0xT7rvyuu8vsaHT6M+dLgHAj1hdss3pEhpgBg7OhpEPAABglU/h4+GHH5bL5fJ6ZGRkeNafPHlSeXl5atu2rWJjYzVq1CiVlZX5vWgAwYXeAaA+n0c+LrnkEu3fv9/z+PDDDz3rpk6dqrfeektLly7VunXrVFJSopEjR/q1YADBid4B4DSfz/lo1aqVUlJSGiwvLy/Xiy++qMWLF2vgwIGSpIULF6pnz57auHGjevfuff7VAgha9A4Ap/k88rFz506lpqaqa9euuvXWW7V3715J0tatW1VTU6OcnBzPthkZGUpLS1NxcfFZ36+qqkoVFRVeDwAtD70DwGk+hY+srCy99NJLWrVqlebPn6/du3erX79+qqysVGlpqSIiItS6dWuv1yQnJ6u0tPSs7zlz5kwlJCR4Hp06dWrSgdhgjMunR1O4XManBxAMQr13APDm09cuQ4YM8fy5V69eysrKUnp6ul577TVFR0c3qYCCggLl5+d7nldUVNBEgBaG3gGgvvOaatu6dWtdeOGF2rVrl1JSUlRdXa0jR454bVNWVtbo97ynRUZGKj4+3usBoGWjdwCh7bzCx9GjR/X111+rQ4cOyszMVHh4uIqKijzrd+zYob179yo7O/u8CwXQctA7gNDm09cu9957r4YNG6b09HSVlJRo+vTpcrvdGj9+vBISEjRp0iTl5+crMTFR8fHxmjx5srKzszlbHQhx9A4A9fkUPv72t79p/PjxOnTokJKSktS3b19t3LhRSUlJkqRZs2YpLCxMo0aNUlVVlXJzczVv3rwmFVZVEaWwmsC6vHoo2/ni1T5t32PSlmaqBMHIZu9A4GjKJd+5JHtocBljAmrKREVFxQ9nrs8tVFg04SNYET6C2ylTo7VaqfLy8qA5l+J07zj8VVfFx7mdLgdNRPgIXr70De7tAgAArCJ8AAAAqwgfAADAKsIHAACwyucby9kSVtlKYTUBW15wa8pl2X28XPzXs3y/PkO3qWe/jweA0ODrDBlOUA1OjHwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsCdjqJ+5hL7lrfZljgXAXm5/rtjD4+bZ/+4IZmqgRAsOD+McGJkQ8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYFbCXV6+NMTJR5txf4PJhW0kygXmJ8VDWbWqx0yUACDJcKj04MfIBAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwK2Nku0d+55I70ZUYKs1eC3f57+jhdQgMdnt7gdAkAfsTqkm1Ol9AoZuH8OEY+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVATvbZeIvVysqNmDLCxhvX9zG6RIABCFmY8BJjHwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsCdjrJa08PljsiyukyAt+/Ol0AgkHC/xQ7XQICTKDeEwWBpblmRTHyAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsCtjZLmPu+RP3djkH3NsFQFNwbxc46bxGPh5//HG5XC5NmTLFs+zkyZPKy8tT27ZtFRsbq1GjRqmsrOx86wTQQtA3ADQ5fGzZskXPPfecevXq5bV86tSpeuutt7R06VKtW7dOJSUlGjly5HkXCiD40TcASE0MH0ePHtWtt96qF154QW3a/GPYv7y8XC+++KKeeeYZDRw4UJmZmVq4cKE2bNigjRs3+q1oAMGHvgHgtCaFj7y8PA0dOlQ5OTley7du3aqamhqv5RkZGUpLS1NxceNXWKyqqlJFRYXXA0DL48++IdE7gGDm8xmdS5Ys0SeffKItW7Y0WFdaWqqIiAi1bt3aa3lycrJKS0sbfb+ZM2eqsLCwwfL/b3sfuS8IrMuru1zGp+2NcTVTJfW83vy7kKROoz+3syO0SP7uG9LZewfOjY3Lq3NSK87Gp5GPffv26Te/+Y0WLVqkqCj/BIOCggKVl5d7Hvv27fPL+wIIDM3RNyR6BxDMfAofW7du1YEDB3TllVeqVatWatWqldatW6c5c+aoVatWSk5OVnV1tY4cOeL1urKyMqWkpDT6npGRkYqPj/d6AGg5mqNvSPQOIJj59LXLoEGD9Pnn3sPvt912mzIyMvTb3/5WnTp1Unh4uIqKijRq1ChJ0o4dO7R3715lZ2f7r2oAQYO+AeBMPoWPuLg4XXrppV7LYmJi1LZtW8/ySZMmKT8/X4mJiYqPj9fkyZOVnZ2t3r17+69qAEGDvgHgTH6/hOisWbMUFhamUaNGqaqqSrm5uZo3b56/dwOgBaFvAKHFZYzxbQpHM6uoqFBCQoJ6/MdjckcG1mwXBL+Oj25wuoSgcMrUaK1Wqry8PGjOpTjdOw5/1VXxcW6ny0ELwqydc+NL3+DGcgAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKr9PtfWX2ihJTHbBj0h/kJkrAHzH7BXnMfIBAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwK3NkuMUYmKqBuOwNfuHz/2XWbsrEZCgHQkjFzJTgx8gEAAKwifAAAAKsIHwAAwCrCBwAAsCpgTzhFkDMupysAAAQoRj4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFUBO9vlwl57FR4T4XQZLVL1gP1OlwAgCHEpc/gLIx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwKqAne3ydXG63FFRTpfRMs3o4nQFfpH+4AanSwBCyuqSbU6X4BfM2nEeIx8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsMqn8DF//nz16tVL8fHxio+PV3Z2tt555x3P+pMnTyovL09t27ZVbGysRo0apbKyMr8XDSC40DsA1OdT+OjYsaMef/xxbd26VR9//LEGDhyo4cOH64svvpAkTZ06VW+99ZaWLl2qdevWqaSkRCNHjmyWwgEED3oHgPpcxhhzPm+QmJio3/3udxo9erSSkpK0ePFijR49WpK0fft29ezZU8XFxerdu/c5vV9FRYUSEhLUddpjckdFnU9paOHSH9zgdAkt1ilTo7VaqfLycsXHxzfLPpqrdxz+qqvi49zNUjNahtzUy5wuoUXypW80+ZyP2tpaLVmyRMeOHVN2dra2bt2qmpoa5eTkeLbJyMhQWlqaiouLz/o+VVVVqqio8HoAaLnoHQBa+fqCzz//XNnZ2Tp58qRiY2O1fPlyXXzxxfrss88UERGh1q1be22fnJys0tLSs77fzJkzVVhY2GB5TUqNaqP57SVY9Zi0xekSEGBs9Q4EN0YlQoPPIx8XXXSRPvvsM23atEl33XWXJkyYoC+//LLJBRQUFKi8vNzz2LdvX5PfC0DgoncAOM3nkY+IiAh1795dkpSZmaktW7bo97//vcaOHavq6modOXLE6zeYsrIypaSknPX9IiMjFRkZ6XvlAIIKvQPAaed9nY+6ujpVVVUpMzNT4eHhKioq8qzbsWOH9u7dq+zs7PPdDYAWht4BhC6fRj4KCgo0ZMgQpaWlqbKyUosXL9batWu1evVqJSQkaNKkScrPz1diYqLi4+M1efJkZWdnn/PZ6gBaJnoHgPp8Ch8HDhzQL3/5S+3fv18JCQnq1auXVq9erV/84heSpFmzZiksLEyjRo1SVVWVcnNzNW/evGYpHEDwoHcAqO+8r/Phb6fn6neaW6iwaK7zEayY7RLcbFznw9+4zkfLwGyX4GXlOh8AAABNQfgAAABWET4AAIBVhA8AAGAV4QMAAFjl8xVObQkvDZc7KtzpMtBE387o0+z74K62QMuzumRbs++DGTXOY+QDAABYRfgAAABWET4AAIBVhA8AAGBVwJ5w2i37W4XHRDhdRotUPWC/0yUACEKcqAl/YeQDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFgVsLNdvvpzmsKiopwuo2Wa1dnpChrVbWqx0yUA+BE2Ln3uK2bgBCdGPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVQE726Uu7pQUfcrpMtBEPSZtcboEAEGI2SuhgZEPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGBVwM52ybvmfUXFBmx5AePti9s4XQKAIMSsEjiJkQ8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABYFbDXL78zYZ/i49xOl2EdlzwG0BT0DgQTRj4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFUBO9sl+9lfyR0Z5XQZ9t3jdAGor8PTG5wuATgnq0u2OV0C6mH20Y9j5AMAAFjlU/iYOXOmrr76asXFxal9+/YaMWKEduzY4bXNyZMnlZeXp7Zt2yo2NlajRo1SWVmZX4sGEFzoHQDq8yl8rFu3Tnl5edq4caPeffdd1dTUaPDgwTp27Jhnm6lTp+qtt97S0qVLtW7dOpWUlGjkyJF+LxxA8KB3AKjPZYwxTX3xwYMH1b59e61bt079+/dXeXm5kpKStHjxYo0ePVqStH37dvXs2VPFxcXq3bv3T75nRUWFEhISlDH5sdA85wMBJVTP+ThlarRWK1VeXq74+Hi/v39z9o7DX3UNyasjI7CE4jkfvvSN8zrno7y8XJKUmJgoSdq6datqamqUk5Pj2SYjI0NpaWkqLi5u9D2qqqpUUVHh9QDQstE7gNDW5NkudXV1mjJliq699lpdeumlkqTS0lJFRESodevWXtsmJyertLS00feZOXOmCgsLGyw/dU2lzAU1TS0PP8Ll8n2wq+OovzRDJQhFzd07EDhC8bd/nJsmj3zk5eXpL3/5i5YsWXJeBRQUFKi8vNzz2Ldv33m9H4DARu8A0KSRj7vvvltvv/221q9fr44dO3qWp6SkqLq6WkeOHPH6DaasrEwpKSmNvldkZKQiIyObUgaAIEPvACD5OPJhjNHdd9+t5cuX6/3331eXLl281mdmZio8PFxFRUWeZTt27NDevXuVnZ3tn4oBBB16B4D6fBr5yMvL0+LFi7Vy5UrFxcV5votNSEhQdHS0EhISNGnSJOXn5ysxMVHx8fGaPHmysrOzz+lsdQAtE70DQH0+hY/58+dLkgYMGOC1fOHChZo4caIkadasWQoLC9OoUaNUVVWl3NxczZs3zy/FAghO9A4A9Z3XdT6aw+m5+p3mFiosmut8BKsek7Y4XQLOQ3Nf56M5cJ2PloEZMsHL2nU+AAAAfEX4AAAAVhE+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFb5dG8Xm9qvdatVOJdIDlaV47kZWDA7VXNSWrrS6TIQglaXbHO6BDRRRWWt2lx4btsy8gEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArArY2S4HBtQqLLrW6TLQRD0mbXG6BJyHU6bG6RIQonJTL3O6BDTRD33jm3PalpEPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWBWwl1cPq2ylsJqALa/5uIxv2xtX89Rxnr6e3dvn13SbsrEZKgEQTFaXbPNpey7HHpwY+QAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVgXsdJK6uFNS9Cmny0AT9Zi0xekSAAQhZq+EBkY+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVATvbpc3WcLkjwp0uA0106M4+TpcQNNo+t8HpEoCA4eu9XUJZMM8MYuQDAABYRfgAAABWET4AAIBVhA8AAGAV4QMAAFgVsLNdDmfWKCza7XQZaCLu7QKgKYJ5BgfOHSMfAADAKp/Dx/r16zVs2DClpqbK5XJpxYoVXuuNMXrooYfUoUMHRUdHKycnRzt37vRXvQCCEH0DQH0+h49jx47psssu09y5cxtd/+STT2rOnDlasGCBNm3apJiYGOXm5urkyZPnXSyA4ETfAFCfz+d8DBkyREOGDGl0nTFGs2fP1oMPPqjhw4dLkl5++WUlJydrxYoVGjdu3PlVCyAo0TcA1OfXcz52796t0tJS5eTkeJYlJCQoKytLxcXFjb6mqqpKFRUVXg8AoaMpfUOidwDBzK+zXUpLSyVJycnJXsuTk5M96840c+ZMFRYWNlgeGX9S7gv8WR1Oc7mMz6/pOOovzVAJ0LS+IZ29dyBwMHMFZ+P4bJeCggKVl5d7Hvv27XO6JABBgN4BBC+/ho+UlBRJUllZmdfysrIyz7ozRUZGKj4+3usBIHQ0pW9I9A4gmPk1fHTp0kUpKSkqKiryLKuoqNCmTZuUnZ3tz10BaCHoG0Do8fmcj6NHj2rXrl2e57t379Znn32mxMREpaWlacqUKZoxY4Z69OihLl26aNq0aUpNTdWIESP8WTeAIELfAFCfz+Hj448/1nXXXed5np+fL0maMGGCXnrpJd133306duyY7rjjDh05ckR9+/bVqlWrFBUV5dN+qiqiFFbj22vQfHa+eLVP23N5ddRnq28gsKwu2ebzazhJNTS4jDG+T31oRhUVFUpISFCnuYUKi6bxBCvCR3A7ZWq0VitVXl4eNOdSnO4dh7/qqvg47gsVrAgfwcuXvuH4bBcAABBaCB8AAMAqwgcAALCK8AEAAKzy6+XV/SnvmvcVFRuw5QWMty9u43QJAIIQJ3bCSYx8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrAnY6ydzNA7m8+rl40ekCGsfl1YHA1pT7rtjALJzQwMgHAACwivABAACsInwAAACrCB8AAMAqwgcAALCK8AEAAKwifAAAAKsIHwAAwCrCBwAAsIrwAQAArArYy6tHxp+U+wKnq2iZXC7j82s6jvpLM1QCoCXjUuk4G0Y+AACAVYQPAABgFeEDAABYRfgAAABWET4AAIBVATvbpUvbvys8JsLpMlqk6gH7nS4BQBBi9gr8hZEPAABgFeEDAABYRfgAAABWET4AAIBVhA8AAGBVwM52KXs9Xe6IKKfLaJnu7OJ0Bain7XMbnC4BOCerS7Y5XQLqCebZR4x8AAAAqwgfAADAKsIHAACwivABAACsInwAAACrAna2y4m2kjvS6SrQ0nR8lJklAHwTzLNKAhUjHwAAwCrCBwAAsIrwAQAArCJ8AAAAqwgfAADAqoCd7RJRwWwX+N+ByX2afR/tn2VGDdCS2LinTajNqGm2kY+5c+eqc+fOioqKUlZWljZv3txcuwLQQtA3gNDQLOHj1VdfVX5+vqZPn65PPvlEl112mXJzc3XgwIHm2B2AFoC+AYSOZgkfzzzzjG6//Xbddtttuvjii7VgwQJdcMEF+u///u/m2B2AFoC+AYQOv5/zUV1dra1bt6qgoMCzLCwsTDk5OSouLm6wfVVVlaqqqjzPy8vLJUm11Sf9XRpgxSlT43QJ5+2UfjgGY4yV/fnaN6Sz946Ko3XNWyzQDEKtb/g9fHz//feqra1VcnKy1/Lk5GRt3769wfYzZ85UYWFhg+XbFz7i79IA+KiyslIJCQnNvh9f+4Z09t6RfuWe5igRaGbfOF2A35xL33B8tktBQYHy8/M9z48cOaL09HTt3bvXStMLJBUVFerUqZP27dun+Ph4p8uxKlSPPVCP2xijyspKpaamOl3KWdE7/iFQ/x41t1A9bikwj92XvuH38NGuXTu53W6VlZV5LS8rK1NKSkqD7SMjIxUZ2XBObUJCQsB8oLbFx8dz7CEmEI/b5n/gvvYNid7RmED8e2RDqB63FHjHfq59w+8nnEZERCgzM1NFRUWeZXV1dSoqKlJ2dra/dwegBaBvAKGlWb52yc/P14QJE3TVVVfpmmuu0ezZs3Xs2DHddtttzbE7AC0AfQMIHc0SPsaOHauDBw/qoYceUmlpqS6//HKtWrWqwclkjYmMjNT06dMbHU5t6Tj20Dv2UD3uxpxP35BC+7MM1WMP1eOWgv/YXcbWXDoAAABxYzkAAGAZ4QMAAFhF+AAAAFYRPgAAgFWEDwAAYFXAhY+5c+eqc+fOioqKUlZWljZv3ux0Sc3u4Ycflsvl8npkZGQ4XZbfrV+/XsOGDVNqaqpcLpdWrFjhtd4Yo4ceekgdOnRQdHS0cnJytHPnTmeK9bOfOvaJEyc2+Dtw/fXXO1NsEKJvtNy+IdE7WmLvCKjw8eqrryo/P1/Tp0/XJ598ossuu0y5ubk6cOCA06U1u0suuUT79+/3PD788EOnS/K7Y8eO6bLLLtPcuXMbXf/kk09qzpw5WrBggTZt2qSYmBjl5ubq5Mngv8PxTx27JF1//fVefwdeeeUVixUGL/pGy+4bEr2jRfYOE0CuueYak5eX53leW1trUlNTzcyZMx2sqvlNnz7dXHbZZU6XYZUks3z5cs/zuro6k5KSYn73u995lh05csRERkaaV155xYEKm8+Zx26MMRMmTDDDhw93pJ5gR98ILfSO5V7LgrV3BMzIR3V1tbZu3aqcnBzPsrCwMOXk5Ki4uNjByuzYuXOnUlNT1bVrV916663au3ev0yVZtXv3bpWWlnr9/BMSEpSVlRUSP39JWrt2rdq3b6+LLrpId911lw4dOuR0SQGPvhHafUOid0jB2TsCJnx8//33qq2tbXAp5eTkZJWWljpUlR1ZWVl66aWXtGrVKs2fP1+7d+9Wv379VFlZ6XRp1pz+GYfiz1/6Ydj05ZdfVlFRkZ544gmtW7dOQ4YMUW1trdOlBTT6Rmj3DYneEay9o1nu7QLfDBkyxPPnXr16KSsrS+np6Xrttdc0adIkByuDLePGjfP8+Wc/+5l69eqlbt26ae3atRo0aJCDlSFQ0TcgBW/vCJiRj3bt2sntdqusrMxreVlZmVJSUhyqyhmtW7fWhRdeqF27djldijWnf8b8/H/QtWtXtWvXLqT+DjQFfeMfQrFvSPSOMwVL7wiY8BEREaHMzEwVFRV5ltXV1amoqEjZ2dkOVmbf0aNH9fXXX6tDhw5Ol2JNly5dlJKS4vXzr6io0KZNm0Lu5y9Jf/vb33To0KGQ+jvQFPSNfwjFviHRO84ULL0joL52yc/P14QJE3TVVVfpmmuu0ezZs3Xs2DHddtttTpfWrO69914NGzZM6enpKikp0fTp0+V2uzV+/HinS/Oro0ePeqXx3bt367PPPlNiYqLS0tI0ZcoUzZgxQz169FCXLl00bdo0paamasSIEc4V7Sc/duyJiYkqLCzUqFGjlJKSoq+//lr33XefunfvrtzcXAerDg70jZbdNyR6R4vsHU5PtznTs88+a9LS0kxERIS55pprzMaNG50uqdmNHTvWdOjQwURERJh/+qd/MmPHjjW7du1yuiy/W7NmjZHU4DFhwgRjzA9T5qZNm2aSk5NNZGSkGTRokNmxY4ezRfvJjx378ePHzeDBg01SUpIJDw836enp5vbbbzelpaVOlx006Bstt28YQ+9oib3DZYwxduMOAAAIZQFzzgcAAAgNhA8AAGAV4QMAAFhF+AAAAFYRPgAAgFWEDwAAYBXhAwAAWEX4AAAAVhE+AACAVYQPAABgFeEDAABY9f8DEry9eu5UGb8AAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"attention_weights = attention_layer.last_attention_weights\n",
"mask=(ex_context_tok != 0).numpy()\n",
"\n",
"plt.subplot(1, 2, 1)\n",
"plt.pcolormesh(mask*attention_weights[:, 0, :])\n",
"plt.title('Attention weights')\n",
"\n",
"plt.subplot(1, 2, 2)\n",
"plt.pcolormesh(mask)\n",
"plt.title('Mask');\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Eil-C_NN1rp"
},
"source": [
"Because of the small-random initialization the attention weights are initially all close to `1/(sequence_length)`. The model will learn to make these less uniform as training progresses."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aQ638eHN4iCK"
},
"source": [
"### The decoder\n",
"\n",
"The decoder's job is to generate predictions for the next token at each location in the target sequence.\n",
"\n",
"1. It looks up embeddings for each token in the target sequence.\n",
"2. It uses an RNN to process the target sequence, and keep track of what it has generated so far.\n",
"3. It uses RNN output as the \"query\" to the attention layer, when attending to the encoder's output.\n",
"4. At each location in the output it predicts the next token.\n",
"\n",
"When training, the model predicts the next word at each location. So it's important that the information only flows in one direction through the model. The decoder uses a unidirectional (not bidirectional) RNN to process the target sequence.\n",
"\n",
"When running inference with this model it produces one word at a time, and those are fed back into the model.\n",
"\n",
"\n",
"\n",
" \n",
" \n",
" \n",
" \n",
"\n",
" A unidirectional RNN \n",
" \n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pZsQJMqNmg_L"
},
"source": [
"Here is the `Decoder` class' initializer. The initializer creates all the necessary layers."
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.633554Z",
"iopub.status.busy": "2023-12-07T12:22:29.632980Z",
"iopub.status.idle": "2023-12-07T12:22:29.639749Z",
"shell.execute_reply": "2023-12-07T12:22:29.639078Z"
},
"id": "erYvHIgAl8kh"
},
"outputs": [],
"source": [
"class Decoder(tf.keras.layers.Layer):\n",
" @classmethod\n",
" def add_method(cls, fun):\n",
" setattr(cls, fun.__name__, fun)\n",
" return fun\n",
"\n",
" def __init__(self, text_processor, units):\n",
" super(Decoder, self).__init__()\n",
" self.text_processor = text_processor\n",
" self.vocab_size = text_processor.vocabulary_size()\n",
" self.word_to_id = tf.keras.layers.StringLookup(\n",
" vocabulary=text_processor.get_vocabulary(),\n",
" mask_token='', oov_token='[UNK]')\n",
" self.id_to_word = tf.keras.layers.StringLookup(\n",
" vocabulary=text_processor.get_vocabulary(),\n",
" mask_token='', oov_token='[UNK]',\n",
" invert=True)\n",
" self.start_token = self.word_to_id('[START]')\n",
" self.end_token = self.word_to_id('[END]')\n",
"\n",
" self.units = units\n",
"\n",
"\n",
" # 1. The embedding layer converts token IDs to vectors\n",
" self.embedding = tf.keras.layers.Embedding(self.vocab_size,\n",
" units, mask_zero=True)\n",
"\n",
" # 2. The RNN keeps track of what's been generated so far.\n",
" self.rnn = tf.keras.layers.GRU(units,\n",
" return_sequences=True,\n",
" return_state=True,\n",
" recurrent_initializer='glorot_uniform')\n",
"\n",
" # 3. The RNN output will be the query for the attention layer.\n",
" self.attention = CrossAttention(units)\n",
"\n",
" # 4. This fully connected layer produces the logits for each\n",
" # output token.\n",
" self.output_layer = tf.keras.layers.Dense(self.vocab_size)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Sd8-nRNzFR8x"
},
"source": [
"#### Training"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UPnaw583CpnY"
},
"source": [
"Next, the `call` method, takes 3 arguments:\n",
"\n",
"* `inputs` - a `context, x` pair where:\n",
" * `context` - is the context from the encoder's output.\n",
" * `x` - is the target sequence input.\n",
"* `state` - Optional, the previous `state` output from the decoder (the internal state of the decoder's RNN). Pass the state from a previous run to continue generating text where you left off.\n",
"* `return_state` - [Default: False] - Set this to `True` to return the RNN state. "
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.643561Z",
"iopub.status.busy": "2023-12-07T12:22:29.642906Z",
"iopub.status.idle": "2023-12-07T12:22:29.648348Z",
"shell.execute_reply": "2023-12-07T12:22:29.647732Z"
},
"id": "PJOi5btHAPNK"
},
"outputs": [],
"source": [
"@Decoder.add_method\n",
"def call(self,\n",
" context, x,\n",
" state=None,\n",
" return_state=False): \n",
" shape_checker = ShapeChecker()\n",
" shape_checker(x, 'batch t')\n",
" shape_checker(context, 'batch s units')\n",
"\n",
" # 1. Lookup the embeddings\n",
" x = self.embedding(x)\n",
" shape_checker(x, 'batch t units')\n",
"\n",
" # 2. Process the target sequence.\n",
" x, state = self.rnn(x, initial_state=state)\n",
" shape_checker(x, 'batch t units')\n",
"\n",
" # 3. Use the RNN output as the query for the attention over the context.\n",
" x = self.attention(x, context)\n",
" self.last_attention_weights = self.attention.last_attention_weights\n",
" shape_checker(x, 'batch t units')\n",
" shape_checker(self.last_attention_weights, 'batch t s')\n",
"\n",
" # Step 4. Generate logit predictions for the next token.\n",
" logits = self.output_layer(x)\n",
" shape_checker(logits, 'batch t target_vocab_size')\n",
"\n",
" if return_state:\n",
" return logits, state\n",
" else:\n",
" return logits"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E1-mLAcUEXpK"
},
"source": [
"That will be sufficient for training. Create an instance of the decoder to test out:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.651960Z",
"iopub.status.busy": "2023-12-07T12:22:29.651328Z",
"iopub.status.idle": "2023-12-07T12:22:29.754850Z",
"shell.execute_reply": "2023-12-07T12:22:29.754189Z"
},
"id": "4ZUMbYXIEVeA"
},
"outputs": [],
"source": [
"decoder = Decoder(target_text_processor, UNITS)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SFWaI4wqzt4t"
},
"source": [
"In training you'll use the decoder like this:\n",
"\n",
"Given the context and target tokens, for each target token it predicts the next target token. "
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.758862Z",
"iopub.status.busy": "2023-12-07T12:22:29.758328Z",
"iopub.status.idle": "2023-12-07T12:22:29.833016Z",
"shell.execute_reply": "2023-12-07T12:22:29.832287Z"
},
"id": "5YM-lD7bzx18"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"encoder output shape: (batch, s, units) (64, 18, 256)\n",
"input target tokens shape: (batch, t) (64, 16)\n",
"logits shape shape: (batch, target_vocabulary_size) (64, 16, 5000)\n"
]
}
],
"source": [
"logits = decoder(ex_context, ex_tar_in)\n",
"\n",
"print(f'encoder output shape: (batch, s, units) {ex_context.shape}')\n",
"print(f'input target tokens shape: (batch, t) {ex_tar_in.shape}')\n",
"print(f'logits shape shape: (batch, target_vocabulary_size) {logits.shape}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zhS_tbk7VQkX"
},
"source": [
"#### Inference\n",
"\n",
"To use it for inference you'll need a couple more methods:"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.836820Z",
"iopub.status.busy": "2023-12-07T12:22:29.836126Z",
"iopub.status.idle": "2023-12-07T12:22:29.840441Z",
"shell.execute_reply": "2023-12-07T12:22:29.839810Z"
},
"id": "SPm12cnIVRQr"
},
"outputs": [],
"source": [
"@Decoder.add_method\n",
"def get_initial_state(self, context):\n",
" batch_size = tf.shape(context)[0]\n",
" start_tokens = tf.fill([batch_size, 1], self.start_token)\n",
" done = tf.zeros([batch_size, 1], dtype=tf.bool)\n",
" embedded = self.embedding(start_tokens)\n",
" return start_tokens, done, self.rnn.get_initial_state(embedded)[0]"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.843716Z",
"iopub.status.busy": "2023-12-07T12:22:29.843193Z",
"iopub.status.idle": "2023-12-07T12:22:29.847217Z",
"shell.execute_reply": "2023-12-07T12:22:29.846553Z"
},
"id": "TzeOhpBvVS5L"
},
"outputs": [],
"source": [
"@Decoder.add_method\n",
"def tokens_to_text(self, tokens):\n",
" words = self.id_to_word(tokens)\n",
" result = tf.strings.reduce_join(words, axis=-1, separator=' ')\n",
" result = tf.strings.regex_replace(result, '^ *\\[START\\] *', '')\n",
" result = tf.strings.regex_replace(result, ' *\\[END\\] *$', '')\n",
" return result"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.850498Z",
"iopub.status.busy": "2023-12-07T12:22:29.850032Z",
"iopub.status.idle": "2023-12-07T12:22:29.854957Z",
"shell.execute_reply": "2023-12-07T12:22:29.854372Z"
},
"id": "v6ildnz_V1MA"
},
"outputs": [],
"source": [
"@Decoder.add_method\n",
"def get_next_token(self, context, next_token, done, state, temperature = 0.0):\n",
" logits, state = self(\n",
" context, next_token,\n",
" state = state,\n",
" return_state=True) \n",
" \n",
" if temperature == 0.0:\n",
" next_token = tf.argmax(logits, axis=-1)\n",
" else:\n",
" logits = logits[:, -1, :]/temperature\n",
" next_token = tf.random.categorical(logits, num_samples=1)\n",
"\n",
" # If a sequence produces an `end_token`, set it `done`\n",
" done = done | (next_token == self.end_token)\n",
" # Once a sequence is done it only produces 0-padding.\n",
" next_token = tf.where(done, tf.constant(0, dtype=tf.int64), next_token)\n",
" \n",
" return next_token, done, state"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9WiXLrVs-FTE"
},
"source": [
"With those extra functions, you can write a generation loop:"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:29.858446Z",
"iopub.status.busy": "2023-12-07T12:22:29.857883Z",
"iopub.status.idle": "2023-12-07T12:22:30.453703Z",
"shell.execute_reply": "2023-12-07T12:22:30.452994Z"
},
"id": "SuehagxL-JBZ"
},
"outputs": [
{
"data": {
"text/plain": [
"array([b'relationship valid neighbor rat richer deposited cafe remarkable hid apology',\n",
" b'trains begun crime quickly stressful physics vanished test wished decisions',\n",
" b'shampoo notice dish products lid ashamed picnic failed stone criminals'],\n",
" dtype=object)"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Setup the loop variables.\n",
"next_token, done, state = decoder.get_initial_state(ex_context)\n",
"tokens = []\n",
"\n",
"for n in range(10):\n",
" # Run one step.\n",
" next_token, done, state = decoder.get_next_token(\n",
" ex_context, next_token, done, state, temperature=1.0)\n",
" # Add the token to the output.\n",
" tokens.append(next_token)\n",
"\n",
"# Stack all the tokens together.\n",
"tokens = tf.concat(tokens, axis=-1) # (batch, t)\n",
"\n",
"# Convert the tokens back to a a string\n",
"result = decoder.tokens_to_text(tokens)\n",
"result[:3].numpy()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5ALTdqCMLGSY"
},
"source": [
"Since the model's untrained, it outputs items from the vocabulary almost uniformly at random."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B6xyru86m914"
},
"source": [
"## The model\n",
"\n",
"Now that you have all the model components, combine them to build the model for training:"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:30.457619Z",
"iopub.status.busy": "2023-12-07T12:22:30.457379Z",
"iopub.status.idle": "2023-12-07T12:22:30.462724Z",
"shell.execute_reply": "2023-12-07T12:22:30.462054Z"
},
"id": "WWIyuy71TkJT"
},
"outputs": [],
"source": [
"class Translator(tf.keras.Model):\n",
" @classmethod\n",
" def add_method(cls, fun):\n",
" setattr(cls, fun.__name__, fun)\n",
" return fun\n",
"\n",
" def __init__(self, units,\n",
" context_text_processor,\n",
" target_text_processor):\n",
" super().__init__()\n",
" # Build the encoder and decoder\n",
" encoder = Encoder(context_text_processor, units)\n",
" decoder = Decoder(target_text_processor, units)\n",
"\n",
" self.encoder = encoder\n",
" self.decoder = decoder\n",
"\n",
" def call(self, inputs):\n",
" context, x = inputs\n",
" context = self.encoder(context)\n",
" logits = self.decoder(context, x)\n",
"\n",
" #TODO(b/250038731): remove this\n",
" try:\n",
" # Delete the keras mask, so keras doesn't scale the loss+accuracy. \n",
" del logits._keras_mask\n",
" except AttributeError:\n",
" pass\n",
"\n",
" return logits"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5rPi0FkS2iA5"
},
"source": [
"During training the model will be used like this:"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:30.465902Z",
"iopub.status.busy": "2023-12-07T12:22:30.465667Z",
"iopub.status.idle": "2023-12-07T12:22:30.671420Z",
"shell.execute_reply": "2023-12-07T12:22:30.670748Z"
},
"id": "8vhjTh84K6Mg"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Context tokens, shape: (batch, s, units) (64, 18)\n",
"Target tokens, shape: (batch, t) (64, 16)\n",
"logits, shape: (batch, t, target_vocabulary_size) (64, 16, 5000)\n"
]
}
],
"source": [
"model = Translator(UNITS, context_text_processor, target_text_processor)\n",
"\n",
"logits = model((ex_context_tok, ex_tar_in))\n",
"\n",
"print(f'Context tokens, shape: (batch, s, units) {ex_context_tok.shape}')\n",
"print(f'Target tokens, shape: (batch, t) {ex_tar_in.shape}')\n",
"print(f'logits, shape: (batch, t, target_vocabulary_size) {logits.shape}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_ch_71VbIRfK"
},
"source": [
"### Train"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8FmzjGmprVmE"
},
"source": [
"For training, you'll want to implement your own masked loss and accuracy functions:"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:30.675550Z",
"iopub.status.busy": "2023-12-07T12:22:30.674981Z",
"iopub.status.idle": "2023-12-07T12:22:30.679248Z",
"shell.execute_reply": "2023-12-07T12:22:30.678679Z"
},
"id": "WmTHr5iV3jFr"
},
"outputs": [],
"source": [
"def masked_loss(y_true, y_pred):\n",
" # Calculate the loss for each item in the batch.\n",
" loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(\n",
" from_logits=True, reduction='none')\n",
" loss = loss_fn(y_true, y_pred)\n",
"\n",
" # Mask off the losses on padding.\n",
" mask = tf.cast(y_true != 0, loss.dtype)\n",
" loss *= mask\n",
"\n",
" # Return the total.\n",
" return tf.reduce_sum(loss)/tf.reduce_sum(mask)"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:30.682724Z",
"iopub.status.busy": "2023-12-07T12:22:30.682192Z",
"iopub.status.idle": "2023-12-07T12:22:30.686290Z",
"shell.execute_reply": "2023-12-07T12:22:30.685639Z"
},
"id": "nRB1CTmQWOIL"
},
"outputs": [],
"source": [
"def masked_acc(y_true, y_pred):\n",
" # Calculate the loss for each item in the batch.\n",
" y_pred = tf.argmax(y_pred, axis=-1)\n",
" y_pred = tf.cast(y_pred, y_true.dtype)\n",
" \n",
" match = tf.cast(y_true == y_pred, tf.float32)\n",
" mask = tf.cast(y_true != 0, tf.float32)\n",
" \n",
" return tf.reduce_sum(match)/tf.reduce_sum(mask)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f32GuAhw2nXm"
},
"source": [
"Configure the model for training:"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:30.689447Z",
"iopub.status.busy": "2023-12-07T12:22:30.689184Z",
"iopub.status.idle": "2023-12-07T12:22:30.717258Z",
"shell.execute_reply": "2023-12-07T12:22:30.716655Z"
},
"id": "9g0DRRvm3l9X"
},
"outputs": [],
"source": [
"model.compile(optimizer='adam',\n",
" loss=masked_loss, \n",
" metrics=[masked_acc, masked_loss])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5DWLI3pssjnx"
},
"source": [
"The model is randomly initialized, and should give roughly uniform output probabilities. So it's easy to predict what the initial values of the metrics should be:"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:30.720596Z",
"iopub.status.busy": "2023-12-07T12:22:30.720355Z",
"iopub.status.idle": "2023-12-07T12:22:30.824151Z",
"shell.execute_reply": "2023-12-07T12:22:30.823460Z"
},
"id": "BuP3_LFENMJG"
},
"outputs": [
{
"data": {
"text/plain": [
"{'expected_loss': 8.517193, 'expected_acc': 0.0002}"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vocab_size = 1.0 * target_text_processor.vocabulary_size()\n",
"\n",
"{\"expected_loss\": tf.math.log(vocab_size).numpy(),\n",
" \"expected_acc\": 1/vocab_size}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "frVba49Usd0Z"
},
"source": [
"That should roughly match the values returned by running a few steps of evaluation:"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:30.828119Z",
"iopub.status.busy": "2023-12-07T12:22:30.827582Z",
"iopub.status.idle": "2023-12-07T12:22:37.523128Z",
"shell.execute_reply": "2023-12-07T12:22:37.522406Z"
},
"id": "8rJITfxEsHKR"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"20/20 [==============================] - 7s 22ms/step - loss: 8.5103 - masked_acc: 2.7872e-04 - masked_loss: 8.5103\n"
]
},
{
"data": {
"text/plain": [
"{'loss': 8.510286331176758,\n",
" 'masked_acc': 0.0002787174307741225,\n",
" 'masked_loss': 8.510286331176758}"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.evaluate(val_ds, steps=20, return_dict=True)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:22:37.526873Z",
"iopub.status.busy": "2023-12-07T12:22:37.526210Z",
"iopub.status.idle": "2023-12-07T12:24:09.857567Z",
"shell.execute_reply": "2023-12-07T12:24:09.856788Z"
},
"id": "BQd_esVVoSf3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/100\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:22:47.016525: W tensorflow/core/common_runtime/type_inference.cc:339] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1:\n",
"type_id: TFT_OPTIONAL\n",
"args {\n",
" type_id: TFT_PRODUCT\n",
" args {\n",
" type_id: TFT_TENSOR\n",
" args {\n",
" type_id: TFT_INT32\n",
" }\n",
" }\n",
"}\n",
" is neither a subtype nor a supertype of the combined inputs preceding it:\n",
"type_id: TFT_OPTIONAL\n",
"args {\n",
" type_id: TFT_PRODUCT\n",
" args {\n",
" type_id: TFT_TENSOR\n",
" args {\n",
" type_id: TFT_INT8\n",
" }\n",
" }\n",
"}\n",
"\n",
"\tfor Tuple type infernce function 0\n",
"\twhile inferring type of node 'cond_35/output/_22'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
"I0000 00:00:1701951767.593696 17028 device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 20s 84ms/step - loss: 5.0340 - masked_acc: 0.2569 - masked_loss: 5.0340 - val_loss: 4.1026 - val_masked_acc: 0.3588 - val_masked_loss: 4.1026\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 2/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 5s 50ms/step - loss: 3.7212 - masked_acc: 0.4018 - masked_loss: 3.7212 - val_loss: 3.3733 - val_masked_acc: 0.4407 - val_masked_loss: 3.3733\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 3/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 4s 36ms/step - loss: 3.1184 - masked_acc: 0.4806 - masked_loss: 3.1184 - val_loss: 2.8760 - val_masked_acc: 0.5082 - val_masked_loss: 2.8760\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 4/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 34ms/step - loss: 2.7033 - masked_acc: 0.5391 - masked_loss: 2.7033 - val_loss: 2.5197 - val_masked_acc: 0.5584 - val_masked_loss: 2.5197\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 5/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 4s 35ms/step - loss: 2.4401 - masked_acc: 0.5739 - masked_loss: 2.4401 - val_loss: 2.2567 - val_masked_acc: 0.6036 - val_masked_loss: 2.2567\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 6/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 35ms/step - loss: 2.1628 - masked_acc: 0.6167 - masked_loss: 2.1628 - val_loss: 2.0215 - val_masked_acc: 0.6413 - val_masked_loss: 2.0215\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 7/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 33ms/step - loss: 2.0348 - masked_acc: 0.6362 - masked_loss: 2.0348 - val_loss: 1.9618 - val_masked_acc: 0.6401 - val_masked_loss: 1.9618\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 8/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 33ms/step - loss: 1.8914 - masked_acc: 0.6545 - masked_loss: 1.8914 - val_loss: 1.8117 - val_masked_acc: 0.6651 - val_masked_loss: 1.8117\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 9/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 30ms/step - loss: 1.7603 - masked_acc: 0.6741 - masked_loss: 1.7603 - val_loss: 1.7312 - val_masked_acc: 0.6810 - val_masked_loss: 1.7312\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 10/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 32ms/step - loss: 1.6979 - masked_acc: 0.6829 - masked_loss: 1.6979 - val_loss: 1.7238 - val_masked_acc: 0.6745 - val_masked_loss: 1.7238\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 11/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 30ms/step - loss: 1.6346 - masked_acc: 0.6898 - masked_loss: 1.6346 - val_loss: 1.5769 - val_masked_acc: 0.6969 - val_masked_loss: 1.5769\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 12/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 30ms/step - loss: 1.5586 - masked_acc: 0.7013 - masked_loss: 1.5586 - val_loss: 1.5677 - val_masked_acc: 0.6931 - val_masked_loss: 1.5677\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 13/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 32ms/step - loss: 1.5177 - masked_acc: 0.7075 - masked_loss: 1.5177 - val_loss: 1.5287 - val_masked_acc: 0.7050 - val_masked_loss: 1.5287\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 14/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 30ms/step - loss: 1.4903 - masked_acc: 0.7107 - masked_loss: 1.4903 - val_loss: 1.4567 - val_masked_acc: 0.7082 - val_masked_loss: 1.4567\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 15/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 31ms/step - loss: 1.4560 - masked_acc: 0.7131 - masked_loss: 1.4522 - val_loss: 1.4132 - val_masked_acc: 0.7213 - val_masked_loss: 1.4132\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 16/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 31ms/step - loss: 1.2320 - masked_acc: 0.7411 - masked_loss: 1.2320 - val_loss: 1.3957 - val_masked_acc: 0.7208 - val_masked_loss: 1.3957\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 17/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 32ms/step - loss: 1.2036 - masked_acc: 0.7467 - masked_loss: 1.2036 - val_loss: 1.3366 - val_masked_acc: 0.7296 - val_masked_loss: 1.3366\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 18/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 31ms/step - loss: 1.2157 - masked_acc: 0.7450 - masked_loss: 1.2157 - val_loss: 1.3473 - val_masked_acc: 0.7329 - val_masked_loss: 1.3473\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 19/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 31ms/step - loss: 1.2205 - masked_acc: 0.7445 - masked_loss: 1.2205 - val_loss: 1.3777 - val_masked_acc: 0.7216 - val_masked_loss: 1.3777\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 20/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 31ms/step - loss: 1.1842 - masked_acc: 0.7523 - masked_loss: 1.1842 - val_loss: 1.2634 - val_masked_acc: 0.7420 - val_masked_loss: 1.2634\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 21/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 30ms/step - loss: 1.2011 - masked_acc: 0.7501 - masked_loss: 1.2011 - val_loss: 1.3571 - val_masked_acc: 0.7303 - val_masked_loss: 1.3571\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 22/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 30ms/step - loss: 1.1825 - masked_acc: 0.7505 - masked_loss: 1.1825 - val_loss: 1.3162 - val_masked_acc: 0.7415 - val_masked_loss: 1.3162\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 23/100\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"100/100 [==============================] - 3s 31ms/step - loss: 1.1826 - masked_acc: 0.7532 - masked_loss: 1.1826 - val_loss: 1.3137 - val_masked_acc: 0.7337 - val_masked_loss: 1.3137\n"
]
}
],
"source": [
"history = model.fit(\n",
" train_ds.repeat(), \n",
" epochs=100,\n",
" steps_per_epoch = 100,\n",
" validation_data=val_ds,\n",
" validation_steps = 20,\n",
" callbacks=[\n",
" tf.keras.callbacks.EarlyStopping(patience=3)])"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:09.861393Z",
"iopub.status.busy": "2023-12-07T12:24:09.861108Z",
"iopub.status.idle": "2023-12-07T12:24:10.028516Z",
"shell.execute_reply": "2023-12-07T12:24:10.027895Z"
},
"id": "38rLdlmtQHCm"
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGwCAYAAACHJU4LAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABRBklEQVR4nO3dd3xUdb7/8de0THoPKRAIofcOUkRRVkDFhh1dUOzg1XW912V3bdd10d29rj93XcuqoCv2FbGBrkiRpkgMRYqUQAIBQkvvM+f3xwkDEdIzmQl5Px+P85iZM98z8wmz47z3e77n+7UYhmEgIiIi4oesvi5AREREpCYKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyW3dcFNIXb7SY7O5uwsDAsFouvyxEREZF6MAyDgoICkpKSsFpr7zNp1UElOzub5ORkX5chIiIijZCVlUWHDh1qbdOqg0pYWBhg/qHh4eE+rkZERETqIz8/n+TkZM/veG1adVA5cbonPDxcQUVERKSVqc+wDQ2mFREREb+loCIiIiJ+S0FFRERE/FarHqMiIiIC4HK5qKio8HUZUsXhcGCz2ZrltRRURESk1TIMg4MHD5Kbm+vrUuRnIiMjSUhIaPI8ZwoqIiLSap0IKe3atSM4OFiTf/oBwzAoLi4mJycHgMTExCa9noKKiIi0Si6XyxNSYmJifF2OnCIoKAiAnJwc2rVr16TTQBpMKyIirdKJMSnBwcE+rkTO5MTn0tSxQwoqIiLSqul0j39qrs9Fp37OIK+kgrTM4wTabYzsou5EERERX/Fpj8pjjz2GxWKptvXs2dOXJQGwIG0ft8xdx0srdvm6FBERkTbN5z0qffr04auvvvI8ttt9XhKDO0UB8ENmLm63gdWqbkUREWk+559/PgMHDuTZZ5/1dSl+z+epwG63k5CQUK+2ZWVllJWVeR7n5+d7paaeCeE47VbySirIOFpEl7hQr7yPiIiI1M7ng2l37NhBUlISqampTJ06lczMzBrbzpkzh4iICM+WnJzslZoC7Fb6d4gAIG3vca+8h4iIiNTNp0FlxIgRzJs3j8WLF/PCCy+QkZHBueeeS0FBwRnbz549m7y8PM+WlZXltdoGdaw6/ZOV67X3EBGR5mMYBsXllT7ZDMNodN3Hjx/nl7/8JVFRUQQHBzNp0iR27NjheX7v3r1MnjyZqKgoQkJC6NOnD59//rnn2KlTpxIXF0dQUBDdunVj7ty5Tf639Cc+PfUzadIkz/3+/fszYsQIOnXqxHvvvceMGTNOa+90OnE6nS1S2+COkYB6VEREWouSChe9H/nCJ++95X8nEBzQuJ/U6dOns2PHDj7++GPCw8N56KGHuPjii9myZQsOh4OZM2dSXl7OihUrCAkJYcuWLYSGmkMSHn74YbZs2cKiRYuIjY1l586dlJSUNOef5nM+H6NyqsjISLp3787OnTt9XYqnR+WnQwUUllUS6vSrfyoRETkLnAgoq1atYtSoUQDMnz+f5ORkPvroI6655hoyMzOZMmUK/fr1AyA1NdVzfGZmJoMGDWLo0KEApKSktPjf4G1+9etbWFjIrl27uPnmm31dCvHhgbSPDGJ/bgkbs3IZ1TXW1yWJiEgtghw2tvzvBJ+9d2Ns3boVu93OiBEjPPtiYmLo0aMHW7duBeC//uu/uPvuu/nyyy8ZP348U6ZMoX///gDcfffdTJkyhbS0NC666CKuuOIKT+A5W/h0jMqDDz7I8uXL2bNnD6tXr+bKK6/EZrNxww03+LIsj4FVp380TkVExP9ZLBaCA+w+2bw5O+5tt93G7t27ufnmm9m0aRNDhw7lb3/7G2AOodi7dy+/+tWvyM7O5sILL+TBBx/0Wi2+4NOgsm/fPm644QZ69OjBtddeS0xMDGvXriUuLs6XZXkMrjr9o3EqIiLiDb169aKyspJvv/3Ws+/o0aNs376d3r17e/YlJydz11138eGHH/LrX/+af/7zn57n4uLimDZtGm+++SbPPvssL7/8cov+Dd7m01M/77zzji/fvk6DT+lRMQxD60mIiEiz6tatG5dffjm33347L730EmFhYfzmN7+hffv2XH755QDcf//9TJo0ie7du3P8+HGWLl1Kr169AHjkkUcYMmQIffr0oaysjE8//dTz3NnC5/Oo+LPeSeEE2KwcKypn79FiX5cjIiJnoblz5zJkyBAuvfRSRo4ciWEYfP755zgcDgBcLhczZ86kV69eTJw4ke7du/OPf/wDgICAAGbPnk3//v0ZO3YsNpvN7zsBGspiNOXibx/Lz88nIiKCvLw8wsPDvfIeV/1jFWmZufz1ugFcOaiDV95DREQarrS0lIyMDDp37kxgYKCvy5Gfqe3zacjvt3pU6jDIM04l17eFiIiItEEKKnUY7JmhVgNqRUREWpqCSh0GVQ2o3XqggOLySt8WIyIi0sYoqNQhKTKIhPBAXG6DTfvyfF2OiIhIm6KgUg8nelXSMnN9WoeIiEhbo6BSD55xKpkapyIiItKSFFTq4dQelVZ8NbeIiEiro6BSD33bR+CwWThSWMa+42fX8tkiIiL+TEGlHgIdNnonmhPSpOn0j4iISItRUKmnQZ5xKrm+LURERNq8lJQUnn322Xq1tVgsfPTRR16tx5sUVOrpxDgVDagVERFpOQoq9XTiyp8fs/MprXD5uBoREZG2QUGlnjpEBREb6qTSbbB5vyZ+ExHxO4YB5UW+2RpwRejLL79MUlISbre72v7LL7+cW2+9lV27dnH55ZcTHx9PaGgow4YN46uvvmq2f6ZNmzZxwQUXEBQURExMDHfccQeFhYWe55ctW8bw4cMJCQkhMjKS0aNHs3fvXgA2bNjAuHHjCAsLIzw8nCFDhvD99983W21nYvfqq59FLBYLgztG8uWWQ6RlHmdoSrSvSxIRkVNVFMMfk3zz3r/NhoCQejW95ppruPfee1m6dCkXXnghAMeOHWPx4sV8/vnnFBYWcvHFF/Pkk0/idDp54403mDx5Mtu3b6djx45NKrOoqIgJEyYwcuRI1q1bR05ODrfddhuzZs1i3rx5VFZWcsUVV3D77bfz9ttvU15eznfffYfFYgFg6tSpDBo0iBdeeAGbzUZ6ejoOh6NJNdVFQaUBBnWM4ssthzSgVkREGi0qKopJkybx1ltveYLKBx98QGxsLOPGjcNqtTJgwABP+yeeeIIFCxbw8ccfM2vWrCa991tvvUVpaSlvvPEGISFmsPr73//O5MmTefrpp3E4HOTl5XHppZfSpUsXAHr16uU5PjMzk//+7/+mZ8+eAHTr1q1J9dSHgkoDDPZM/HYcwzA8CVNERPyAI9js2fDVezfA1KlTuf322/nHP/6B0+lk/vz5XH/99VitVgoLC3nsscf47LPPOHDgAJWVlZSUlJCZmdnkMrdu3cqAAQM8IQVg9OjRuN1utm/fztixY5k+fToTJkzgF7/4BePHj+faa68lMTERgAceeIDbbruNf/3rX4wfP55rrrnGE2i8RWNUGqBfhwhsVguH8ss4kFfq63JERORUFot5+sUXWwP/j+vkyZMxDIPPPvuMrKwsvvnmG6ZOnQrAgw8+yIIFC/jjH//IN998Q3p6Ov369aO8vNwb/2qnmTt3LmvWrGHUqFG8++67dO/enbVr1wLw2GOP8eOPP3LJJZfw9ddf07t3bxYsWODVehRUGiA4wE6vxDBAE7+JiEjjBQYGctVVVzF//nzefvttevToweDBgwFYtWoV06dP58orr6Rfv34kJCSwZ8+eZnnfXr16sWHDBoqKijz7Vq1ahdVqpUePHp59gwYNYvbs2axevZq+ffvy1ltveZ7r3r07v/rVr/jyyy+56qqrmDt3brPUVhMFlQYalKyJ30REpOmmTp3KZ599xmuvvebpTQFz3MeHH35Ieno6GzZs4MYbbzztCqGmvGdgYCDTpk1j8+bNLF26lHvvvZebb76Z+Ph4MjIymD17NmvWrGHv3r18+eWX7Nixg169elFSUsKsWbNYtmwZe/fuZdWqVaxbt67aGBZv0BiVBhrcKZJ/rd2rHhUREWmSCy64gOjoaLZv386NN97o2f/MM89w6623MmrUKGJjY3nooYfIz89vlvcMDg7miy++4L777mPYsGEEBwczZcoUnnnmGc/z27Zt4/XXX+fo0aMkJiYyc+ZM7rzzTiorKzl69Ci//OUvOXToELGxsVx11VU8/vjjzVJbTSxGK14OOD8/n4iICPLy8ggPD2+R99xzpIjz/7KMAJuVTY9fhNNua5H3FRGR6kpLS8nIyKBz584EBgb6uhz5mdo+n4b8fuvUTwN1igkmOiSAcpebH7ObJ+GKiIjImSmoNJDFYmFQciSgcSoiIuJb8+fPJzQ09Ixbnz59fF1es9AYlUYY3CmKJdtySMs8zgw6+7ocERFpoy677DJGjBhxxue8PWNsS1FQaYQTPSrp6lEREfG5VjzUssnCwsIICwvzdRln1Fyfi079NEL/5EisFtifW8KhfE38JiLiCyd6DIqLi31ciZzJic+lqT076lFphFCnne7xYWw7WMAPmceZ2DfR1yWJiLQ5NpuNyMhIcnJyAPPSWi1t4nuGYVBcXExOTg6RkZHYbE27OlZBpZEGd4pi28EC0jJzFVRERHwkISEBwBNWxH9ERkZ6Pp+mUFBppEHJkbz1bSY/aOI3ERGfsVgsJCYm0q5dOyoqKnxdjlRxOBxN7kk5QUGlkQZ3MqfS37gvj/JKNwF2DfcREfEVm83WbD+M4l/069pIqbEhRAQ5KKt0s+2gJn4TERHxBgWVRrJYLAzqGAlA2l6d/hEREfEGBZUmGNyxaiXlrFzfFiIiInKWUlBpAk+PigbUioiIeIWCShMMSI7EYoGsYyUcLijzdTkiIiJnHQWVJggPdNCtXSiALlMWERHxAgWVJtI4FREREe9RUGkiXfkjIiLiPQoqTXSiR2XjvjwqXW4fVyMiInJ2UVBpoi5xoYQF2impcLHtYIGvyxERETmrKKg0kdVqYWByJKBxKiIiIs1NQaUZDDoxoFbjVERERJqVgkozGFw1oFY9KiIiIs1LQaUZDEo2e1QyjhRxrKjcx9WIiIicPRRUmkFEsIMucSEApGfp9I+IiEhzUVBpJifGqaTtzfVtISIiImcRBZVmcnKGWvWoiIiINBcFlWZyYoba9MxcXG7Dt8WIiIicJRRUmkn3+DBCAmwUlbvYkaOJ30RERJqDgkozsVktDKia+E3jVERERJqHgkoz8oxTydQ4FRERkeagoNKMPCspK6iIiIg0CwWVMyk+Bmueh22fNeiwE5co7zpcRF5xhTcqExERaVMUVM5k/Tz44rew8q8NOiw6JICUmGBAlymLiIg0BwWVMxk4Fax22LcODv3YoENPjlPJ9UJhIiIibYuCypmExUOPi837619v0KEapyIiItJ8FFRqMmSaebvxHSgvrvdhJ8appGfl4tbEbyIiIk3iN0HlqaeewmKxcP/99/u6FFPqBRDREUrzYMvCeh/WMyGMIIeNgtJKdh0u9GKBIiIiZz+/CCrr1q3jpZdeon///r4u5SSrFYb80ryfVv/TP3ablf4dIszDdPpHRESkSXweVAoLC5k6dSr//Oc/iYqK8nU51Q28CSw2yFwDOdvqfdggDagVERFpFj4PKjNnzuSSSy5h/PjxdbYtKysjPz+/2uZV4YnQfaJ5vwG9KoM1oFZERKRZ+DSovPPOO6SlpTFnzpx6tZ8zZw4RERGeLTk52csVAkOmm7cb3oaK0nodcqJHZUdOIfmlmvhNRESksXwWVLKysrjvvvuYP38+gYGB9Tpm9uzZ5OXlebasrCwvVwl0vRDCO0DJcdj6Sb0OiQtzkhwdhGHAhqxc79YnIiJyFvNZUFm/fj05OTkMHjwYu92O3W5n+fLlPPfcc9jtdlwu12nHOJ1OwsPDq21eZ7XB4KpBtevn1fuwQckapyIiItJUPgsqF154IZs2bSI9Pd2zDR06lKlTp5Keno7NZvNVaacbdBNYrLB3JRzZUa9DNE5FRESk6ey+euOwsDD69u1bbV9ISAgxMTGn7fe5iPbQ7SL4abHZqzLhyToPGdzpZI+KYRhYLBYvFykiInL28flVP63GqYNqK8vqbN4zIRyn3UpeSQW7jxR5tzYREZGzlM96VM5k2bJlvi6hZl1/AWFJUJAN2z6FvlNqbR5gNyd+W7fnOD9k5tIlLrSFChURETl7qEelvmx2c6wK1HtQ7YnLlDVORUREpHEUVBpi8M2ABTJWwNFddTevGlCrK39EREQaR0GlISI7QteqGXTT3qiz+Ykele0H8yksq/RmZSIiImclBZWGOjGoNn0+VJbX2jQ+PJD2kUG4Ddi4L9frpYmIiJxtFFQaqvsECI2HosOw/fM6mw/U6R8REZFGU1BpKJvj5KDaeixUONizkrIG1IqIiDSUgkpjnJhSf9fXcHxPrU0HeWaoNSd+ExERkfpTUGmMqBRIHWfer2NQbZ+kcAJsVo4VlZN5rNj7tYmIiJxFFFQa68Sg2h/eBFdFjc2cdht92puLJ2o+FRERkYZRUGmsHhdDSBwUHoKfvqi16clxKrktUJiIiMjZQ0GlsewBMHCqeb+OmWoHaSVlERGRRlFQaYoTg2p3fgW5mTU3q+pR2XqggJJyV0tUJiIiclZQUGmKmC7QeSxgmGNVapAYEUhCeCAut8Hyn3Jarj4REZFWTkGlqU4Mqk37F7jOPE2+xWLhqsHtAfjHsl26TFlERKSeFFSaquelEBwDBdmw8z81NpsxpjOBDisb9+XxzY4jLVigiIhI66Wg0lR2Jwy4wbxfy6DamFAnNwzvCMDfl+5sgcJERERaPwWV5nDi9M+OLyFvf43N7hibSoDNyncZx1i351jL1CYiItKKKag0h9hu0GkMGO46BtUGMWVIBwD+/rV6VUREROqioNJcPINq3wB3zZcg331eF2xWC8t/OsymfXktU5uIiEgrpaDSXHpNhqAoyN8HO5fU2KxjTDCXDUgC4HmNVREREamVgkpzcQSeHFSb9nqtTe85vwsAi388yE+HCrxdmYiISKuloNKcBk8zb7cvgvwDNTbrFh/GxD4JAPxDvSoiIiI1UlBpTu16QseRYLggveZBtQAzx3UF4OMN2ew9WtQS1YmIiLQ6CirN7USvStob4HbX2KxfhwjO6x6H24AXl+9qoeJERERaFwWV5tbnCgiMMBcp3L201qazLjB7VT5Yv48DeSUtUJyIiEjroqDS3BxB0P96834tM9UCDEuJZnjnaCpcBi+v2O392kRERFoZBRVvGHJiUO3nUFj7asmzqsaqvP1dJkcKy7xdmYiISKuioOIN8X2gwzBwV0L6/FqbntstlgEdIiitcPPayowWKlBERKR1UFDxlhMz1a5/vdZBtRaLxXMF0Btr9pJXXNECxYmIiLQOCire0udKcIbD8QzYs6LWpuN7xdMjPozCskpeX7OnZeoTERFpBRRUvCUgBPpdY96vY1Ct1WrhnnHmbLWvrcqgqKzSy8WJiIi0Dgoq3nTi9M/WT6HoSK1NL+2fREpMMLnFFbz1bab3axMREWkFFFS8KbE/JA0GdwWkv1VrU5vVwt1VawC9/M1uSitqXoFZRESkrVBQ8TbPoNp5YBi1Nr1yUAeSIgI5XFDG++v3eb00ERERf6eg4m19p0BAKBzbBXtW1to0wG7ljrGpALy4bBcVrpqvFhIREWkLFFS8zRkK/a4276e9Xmfz64d3JDY0gP25JSxMz/ZycSIiIv5NQaUlnDj9s2VhnYNqAx02Zowxe1X+sWwnLnftp4tERETOZgoqLSFpkDmo1lUOS/9YZ/ObzulIRJCD3YeLWLz5YAsUKCIi4p8UVFrKRU+Yt+vnwsFNtTYNC3QwfVQKAH9fuhOjjkG4IiIiZysFlZaSMgZ6XwGGGxb9ps4rgG4ZnUJIgI2tB/L5elvtCxuKiIicrRRUWtJFT4A9EPauhB8X1No0MjiAm87pBKhXRURE2i4FlZYU2RHG/Mq8/+XDUF5ca/MZ53YmwG7lh8xc1uw62gIFioiI+BcFlZY26r8gIhny98Gq/1dr03ZhgVw/LBkwe1VERETaGgWVlhYQfHJg7apnIbf2dX3uPK8LdquF1buOkpZ53Pv1iYiI+BEFFV/ofQV0GgOVpeYpoFq0jwziykHtAXj+a/WqiIhI26Kg4gsWC0x6GixW2PIRZKyotfnd53fBaoEl23LYkp3fMjWKiIj4AQUVX0noC0NvNe8v+g24KmtsmhoXysX9EgF4fpl6VUREpO1QUPGlcb+DwEjI+dGcCK4WM8d1BeDzTQfYmVPYAsWJiIj4noKKLwVHwwW/N+8vfRKKj9XYtFdiOON7xWMY8MKyXS1UoIiIiG8pqPjakFugXR8oOW6GlVrMusDsVfkofT9Zx2qfg0VERORsoKDiazY7THrKvP/9a3Bwc41NByZHMqZrLC63wUsr1KsiIiJnPwUVf9B5LPS+3FwHaHHt6wCdGKvy3vf7yMkvbakKRUREfEJBxV9c9AdzHaA938CWhTU2Oyc1miGdoiivdPPPb3a3YIEiIiItT0HFX0R2hNH3mfe//H2N6wBZLBZmVfWqzP82k+NF5S1VoYiISItTUPEno++H8A6QlwWrn6ux2fk94uiTFE5xuYtXVqpXRUREzl4KKv4kIBgu+l/z/spnITfrjM0sFgv3XtANgJdX7Gbz/rwWKlBERKRlKaj4mz5XQafRUFkC/6l5HaAJfeKZ2CeBCpfB/e+mU1rhasEiRUREWoaCir85dR2gHxfAnpU1NLPwx6v6ERfmZGdOIU8t2tbChYqIiHifgoo/SugHQ6ab9xc9VOM6QNEhAfzp6v4AzFu9h292HG6hAkVERFqGgoq/Gvd7CIyAQ5shbV7NzXq04+ZzOgHw4PsbyC3WVUAiInL28GlQeeGFF+jfvz/h4eGEh4czcuRIFi1a5MuS/EdIjLloIcDXf6h1HaDfXtyL1NgQDuWX8buPNmPUMmGciIhIa+LToNKhQweeeuop1q9fz/fff88FF1zA5Zdfzo8//ujLsvzH0BkQ18tcB2jZnBqbBQXY+Ot1A7FbLXy28QAL07NbsEgRERHv8WlQmTx5MhdffDHdunWje/fuPPnkk4SGhrJ27VpfluU/bHZzYC3AulfgUM0BbkByJP91oXnJ8sMLN7M/t6QlKhQREfEqvxmj4nK5eOeddygqKmLkyJFnbFNWVkZ+fn617ayXeh70mmyuA7TooVrXAbrn/C4M6hhJQWklv34vHbdbp4BERKR1a1RQKSoq4uGHH2bUqFF07dqV1NTUaltDbNq0idDQUJxOJ3fddRcLFiygd+/eZ2w7Z84cIiIiPFtycnJjym99LvoD2JzmOkBbP66xmd1m5a/XDiQ4wMba3cc0a62IiLR6FqMRIy9vuOEGli9fzs0330xiYiIWi6Xa8/fdd1+9X6u8vJzMzEzy8vL44IMPeOWVV1i+fPkZw0pZWRllZWWex/n5+SQnJ5OXl0d4eHhD/4zW5es/wIo/Q0RHmPUdOIJqbPr2d5nM/nATATYrC2eNplfiWf5vIyIirUp+fj4RERH1+v1uVFCJjIzks88+Y/To0Y0usibjx4+nS5cuvPTSS3W2bcgf2uqVF8Hfh0H+fvNqoPP+p8amhmFw+xvr+WrrIXomhPHRzNEEOmwtWKyIiEjNGvL73ahTP1FRUURHRzequLq43e5qvSZSJSAEflG1DtA3z9S4DhCYs9Y+NaUfMSEBbDtYwP99ub2FihQREWlejQoqTzzxBI888gjFxcVNevPZs2ezYsUK9uzZw6ZNm5g9ezbLli1j6tSpTXrds1bfKdBxVNU6QI/U2jQ21MnTU8xZa19ZmcHqXUdaokIREZFm1ahTP4MGDWLXrl0YhkFKSgoOh6Pa82lpafV6nRkzZrBkyRIOHDhAREQE/fv356GHHuIXv/hFvY5vU6d+TjiwAV46DzBg+ueQUvvpt9kfbuTt77JIighk0f1jiQhy1NpeRETE2xry+21vzBtcccUVjTnsNK+++mqzvE6bkjgAhkyD9fPMy5XvXA7Wmsef/P6S3qzZdZQ9R4t5dOFmnr1+UMvVKiIi0kSN6lHxF22yRwWg6Aj8bTCU5sHFf4Hht9faPC3zOFe/sBq3AX+7YRCTByS1UKEiIiKn8/pgWoDc3FxeeeUVZs+ezbFj5jo0aWlp7N+/v7EvKfUVEgvn/9a8/+XvIfuHWpsP7hjFrHFdAfjdgk0czCv1doUiIiLNolFBZePGjXTv3p2nn36av/zlL+Tm5gLw4YcfMnv27OasT2oy/HbodhFUlsI7U6Ewp9bm917YjQEdIsgvreTB9zdo1loREWkVGhVUHnjgAaZPn86OHTsIDAz07L/44otZsWJFsxUntbDaYMorENPNnFvl3ZuhsrzG5g6blWeuG0igw8rKnUeYt3pPy9UqIiLSSI0KKuvWrePOO+88bX/79u05ePBgk4uSegqMgBveBmcEZK2Fzx+sdS2gLnGh/O7iXgA8tXgbPx0qaKlKRUREGqVRQcXpdJ5xQcCffvqJuLi4JhclDRDbDa5+FbBA2uvmKsu1uOmcTpzfI47ySjf3v5NOeaW7ZeoUERFphEYFlcsuu4z//d//paKiAjBnQs3MzOShhx5iypQpzVqg1EO3X8D4x8z7i38DGd/U2NRisfCnKf2JCnaw5UA+f/3qp5apUUREpBEaFVT+7//+j8LCQtq1a0dJSQnnnXceXbt2JSwsjCeffLK5a5T6GH0f9L0a3JXw/jQ4vrfGpu3CA5lzVT8AXly+i+8yjrVUlSIiIg3SpHlUVq1axYYNGygsLGTw4MGMHz8ewzBOW03ZW9rsPCo1KS+GuRPN2Wvj+8KML801gmrw4Psb+GD9PjpEBbHovnMJC9SstSIi4n1en0flz3/+MwCjR4/mnnvu4X/+538YP348LpeLG2+8sTEvKc0hIBiufwtC4uDQZvjo7loH1z46uTcdooLYd7yExz/Z0oKFioiI1E+jg8rPp793uVxcf/31pKenN0dd0lgRHeDaf4HVAVsWwjd/qbFpWKCDv143EIsFPli/j8WbD7RgoSIiInVrVFD57LPPePDBB/nggw8AqKys5JprruHHH39k6dKlzVqgNEKnkXBJVUD5+g+w7fMamw5Lieau87oAMPvDTeTka9ZaERHxH40KKsOGDePf//43t956Kx9//DFTpkxh+/btLF26lISEhOauURpjyHQYVrUG0Id3QM62Gpv+anx3+iSFc7y4gv/+YKNmrRUREb/R6LV+LrjgAt544w2mTJlCRkYGy5cvJz4+vjlrk6aaOAc6jYHyAnj7eig+89U9AXYrz143EKfdyvKfDnP/u5pfRURE/EO9r/q56qqrzrh/7dq1dO3aldjYWM++Dz/8sHmqq4Ou+qmHoiPw8jjIy4TUcTD1A7DZz9j0kw3Z/OrddCrdBud2i+WFm4YQ6jxzWxERkcbyylU/ERERZ9wmTJhAly5dqu0TPxISCze8BY5g2L0Uvnq0xqaTByTx6vRhBAfY+GbHEW7851qOFpa1YLEiIiLVNWkeFV9Tj0oD/LgA3p9u3r/iRRh4Q41N07NyuWXudxwvrqBzbAhv3Dqc5OjglqlTRETOel6fR+WEw4cPs3LlSlauXMnhw4eb8lLibX2uhLH/bd7/5D7Yt77GpgOTI/ng7lG0jwwi40gRU15YzdYDp6/tJCIi4m2NCipFRUXceuutJCYmMnbsWMaOHUtSUhIzZsyguLi4uWuU5nL+b6HHxeAqg3enQkHNK113iQvl33ePokd8GDkFZVz70hq+3X20BYsVERFpZFB54IEHWL58OZ988gm5ubnk5uaycOFCli9fzq9//evmrlGai9UKV74EcT2h4AC8exNU1jwGJSEikPfuHMmwlCgKSiu5+bXv+OLHmsONiIhIc2vUGJXY2Fg++OADzj///Gr7ly5dyrXXXttip4E0RqWRju6Cf46D0jwYeBNc/neoZX2m0goXs976ga+2HsJqgSev7McNwzu2YMEiInI28foYleLi4jPOmdKuXTud+mkNYrrA1XPBYoX0N+Hbl2ptHuiw8eJNg7luaDJuw5zB9m9LdtCKx2GLiEgr0aigMnLkSB599FFKS09Ot15SUsLjjz/OyJEjm6048aKuF8IvnjDvf/Fb2L2s1uZ2m5WnpvRj1riuAPzff37isY9/1Cy2IiLiVY069bNp0yYmTpxIWVkZAwYMAGDDhg0EBgbyxRdf0KdPn2Yv9Ex06qeJDMNcYXnD2xAUBbcvhejOdR42b1UGj3+6BcOAS/on8sy1A3DabS1QsIiInA0a8vvd6HlUiouLmT9/Ptu2mWvI9OrVi6lTpxIUFNSYl2sUBZVmUFEKcydBdhq06w0zvgRnWJ2Hfbwhm1+/l06Fy2B01xheunmoZrEVEZF68XpQWbFiBaNGjcJur/7DVFlZyerVqxk7dmxDX7JRFFSaSX62Oc1+4UHoeSlc+wZY6+4hWbnjCHf+63uKyl30bR/OvFuGExvqbIGCRUSkNfP6YNpx48Zx7NjpC9zl5eUxbty4xryk+FJ4Elz3JtgCYNun8MEttV62fMKYbrG8fcc5xIQEsHl/Ple/sJrMoxpMLSIizadRQcUwDCxnuJz16NGjhISENLko8YHkYTDlFTOsbFkIb06B0rpno+3fwZzFtkNUEHuOFjPlxdX8mJ3XAgWLiEhb0KBTPydWUF64cCETJ07E6TzZze9yudi4cSM9evRg8eLFzV/pGejUjxfsXg7vTIXyAkjoB1P/DWGnX4r+czn5pfzyte/YdrCAMKedl385lJFdYlqgYBERaW28durnxOrIhmEQFhZWbcXkhIQE7rjjDt58880mFS8+lnoeTP8UQuLg4CZ47SI4trvOw9qFB/LeXSMZ0TmagrJKpr32HYs2HWiBgkVE5GzWoB6V4uJigoODefzxx3nwwQd9fppHPSpedHQXvHkVHN9jhpab/g2JA+o8rLTCxf3vpLP4x4NYLPDE5X256ZxO3q9XRERaDa/1qMTGxnLppZeSmJhIQUFBk4oUPxfTBW790jz9U3QY5l5inhaqQ6DDxvNTB3PD8I4YBvz+o808p1lsRUSkkRoUVLZu3cqECRN47733SElJYcSIETz55JNs2rTJW/WJL4XFw/TPIeVcc8zK/KvhxwV1HmazWvjjlX35rwu7AfDMf37i6cXbFVZERKTBGj3hW15eHp9//jkLFy5k8eLFREdHc9lll3HZZZdx3nnnYbN5f6ZSnfppIRWlsOAO82ogLHDxn2H47fU69NWVGTzx6RYApo3sxKOT+2C11rwAooiInP28Po8KmANrb7jhBt555x0OHz7MSy+9hMvl4pZbbiEuLo758+c39qXF3zgCzUUMh84ADPj8Qfj6SXMK/jrMGNOZP17ZD4sFXl+zl998uBGX1gcSEZF6anSPSm1++OEHKisrGTZsWHO/dDXqUWlhhgHL/wTL/mg+HjIdLnmmXrPYLvhhH79+bwNuAyYPSOKZawfgsDU6J4uISCvmtR6VP/3pT5SUlHger1q1irKykzOYFhQUcM899zBo0CCvhxTxAYsFzn8ILv0rWKywfh6890vz1FAdrhzUgedvHIzDZuGTDdncMz+NskqX92sWEZFWrUE9KjabjQMHDtCuXTsAwsPDSU9PJzU1FYBDhw6RlJSEy9UyP0DqUfGhLR/Dv28DVxl0Gg3XvwVBkXUetnRbDne+uZ7ySjfndovl5ZuHEhSglZdFRNoSr/Wo/DzT6CqONqz3ZXDzh+AMh72rYN4lUHCwzsPG9WzHvOnDCA6w8c2OI0yb+x2FZZUtULCIiLRGGiQgjZcyBm75HELj4dBmePUX5kRxdRjVNZZ/zRhOmNPOdxnHmPrKt+QVV7RAwSIi0tooqEjTJPSDGV9CdCrkZsKrF8H+tDoPG9IpmrduP4fIYAcbsnK5/p9rOVJY94rNIiLSttgbesArr7xCaGgoAJWVlcybN4/Y2FgAzVbbVkWlmLPYzr8aDqTD65Phun9BlwtqPaxfhwjevWMkU1/5lq0H8rnupTXMv+0cEiICW6RsERHxfw0aTJuSkoLFUvdkXRkZGU0qqr40mNbPlBXAuzfB7mVgdcCVL0K/q+s8bPfhQqa+8i0H8krpGB3M/NtGkBwd7P16RUTEJxry++2VeVRaioKKH6osh4/ugs3/Nh9PfBrOuavOw7KOFTP1lW/JPFZMUkQgb942gtS4UC8XKyIivuC1q36+/vprevfuTX5+/mnP5eXl0adPH7755puGVStnF3sAXPUKDL/TfLz4Ifjy9+Cu/ZL15Ohg3rtzJF3iQsjOK+Xal9ay/aBOJYqItHUNCirPPvsst99++xnTT0REBHfeeSfPPPNMsxUnrZTVCpOehgsfMR+v/hu8dS2UHK/1sISIQN69cyS9EsM5UljG9S+vYdO+vBYoWERE/FWDgsqGDRuYOHFijc9fdNFFrF+/vslFyVnAYoFzfw1Xvwb2INj5FfzzQji8vdbDYkOdvHP7OQxIjuR4cQU3/nMt6/cea6GiRUTE3zQoqBw6dAiHw1Hj83a7ncOHDze5KDmL9J1iXr4c0RGO7TLDyvZFtR4SEezgzRnDGZ4STUFZJTe/+h2rdx5poYJFRMSfNCiotG/fns2bN9f4/MaNG0lMTGxyUXKWSewPdyyFTmOgvADevgFW/KXW1ZfDAh28futwzu0WS3G5i+nz1vH1tkMtWLSIiPiDBgWViy++mIcffpjS0tMXoSspKeHRRx/l0ksvbbbi5CwSEgu//AiG3Q4Y8PUT8P50KC+q8ZCgABuvTBvK+F7xlFe6ufNf61m06UBLVSwiIn6gQZcnHzp0iMGDB2Oz2Zg1axY9evQAYNu2bTz//PO4XC7S0tKIj4/3WsGn0uXJrdT61+GzX4O7AuL7wvXzzUnjalDhcvOrd9P5dOMBrBb489UDmDKkQ8vVKyIizcqr86js3buXu+++my+++MKzKKHFYmHChAk8//zzdO7cufGVN5CCSiuWuRbevRmKciAoGq59HTqPrbG5y23wm39v5P31+wAY1DGSGWM6M7FPAnabVoIQEWlNWmTCt+PHj7Nz504Mw6Bbt25ERUU1qtimUFBp5fL2w7tTIfsHsNhg4hwYfod5xdAZuN0Gf/5yO69+k0G5yw1A+8ggpo3qxHXDOhIRVPNAbxER8R+amVZaj4oS+OQ+2Piu+XjQTXDJM2B31nhITkEpb67NZP7avRwtKgcgJMDGNUOTuWV0Cp1iQlqichERaSQFFWldDAPW/B3+8wgYbugw3FzUMCyh1sNKK1wsTN/Pqysz+OlQIWB2xvyiVzwzxnRmeOfoeq1NJSIiLUtBRVqnnUvgg1ugNA/CEuG6+dBhSJ2HGYbByp1HeOWbDJb/dHIen77tw5kxpjOX9EsiwK5xLCIi/kJBRVqvo7vgnRvh8DawOWHy/4OBN9T78J05Bby6cg8fpu2jrNIcx9IuzMm0USncOLwjUSEB3qpcRETqSUFFWreyAvjwTtj+mfn4nHvgF0+AzV7vlzhWVM5b3+7ljTV7ySkoAyDQYeWqwR24dXRnurbTyswiIr6ioCKtn9sNy5+C5U+bjzufB9fMg+DoBr1MeaWbTzdm8+rKDH7MPrnq9/k94pgxpjNjusZqHIuISAtTUJGzx5aFsOBuqCgyJ4W7/m2I793glzEMg28zjvHqygy+2nrIM3t/j/gwbh2TwuUD2xPosDVv7SIickatJqjMmTOHDz/8kG3bthEUFMSoUaN4+umnPTPe1kVBpY049KO5PlDuXnCEmONWel4CAcGNerk9R4qYuyqD99fvo7jcBUB8uJO7zuvCDcM7KrCIiHhZqwkqEydO5Prrr2fYsGFUVlby29/+ls2bN7NlyxZCQuqeC0NBpQ0pPgbvT4OMFeZjiw3a9YKkgZA0GNoPhnZ9wF7/wbJ5JRW8uy6Tuav2cCDPXL8qLswMLDcO70hQgAKLiIg3tJqg8nOHDx+mXbt2LF++nLFjT59OvaysjLKyMs/j/Px8kpOTFVTaClclLH0S0t+CwoOnP28LMNcOaj/4ZHiJ7Q7W2gNHWaWL97/fxwvLdrE/twSA2FAnd45NZeo5HQkOqP8gXhERqVurDSo7d+6kW7dubNq0ib59+572/GOPPcbjjz9+2n4FlTYoPxv2p5nT72enmfdLc09v5wiBxAFV4WWQuUWnnnGa/vJKN/9O28fzS3ey77gZWGJCArh9bCo3n9OJEKcCi4hIc2iVQcXtdnPZZZeRm5vLypUrz9hGPSpSI8OA4xlmcPEEmHRzEO7PBUaagcUTXgZDRHvP0xUuNwvS9vP3pTvJPFYMQHRIALed25lfjkwhVIFFRKRJWmVQufvuu1m0aBErV66kQ4cO9TpGY1SkVm4XHPnplPCSBgc3gav89LadRsMFv4dOozy7KlxuPvphP88v3cmeo2ZgiQx2cNuYzkwblUJYoBZBFBFpjFYXVGbNmsXChQtZsWIFnTt3rvdxCirSYJXlkLPl5Omi7B8gZysY5tU/dLkQLvgdtD85dX+ly83HG7L5+9c72X3E7KEJD7QzY0wq00enaNVmEZEGajVBxTAM7r33XhYsWMCyZcvo1q1bg45XUJFmkbcPVvwFfvgXuCvNfT0vhXG/qzZni8tt8OnGbJ5bsoNdh83AEhZo59bRnbl1dGcighVYRETqo9UElXvuuYe33nqLhQsXVps7JSIigqCgoDqPV1CRZnVsNyx7Gja+CxiABfpOgXG/hZgunmYut8Fnmw7wtyU72JFjrtoc5rQzfXQKM8Z0JjJY6wmJiNSm1QSVmqYunzt3LtOnT6/zeAUV8YqcbbBsDmz5yHxssZkLI573EER29DRzuw0WbT7Ic0t2sP1QAQAhATamjUrhtnNTidYCiCIiZ9RqgkpTKaiIVx3YAF8/CTu+MB9bHTD0Fjj31xCW4Gnmdht8ueUg/2/JTrYeMNcTCnLYuG5YMreO7kzHmMbNoCsicrZSUBFpTlnfwdd/gIzl5mN7EAy/Hcb8qtoiiW63wVdbD/Hc1zvYvN8MLFYLTOqbyG3ndmZQxyhfVC8i4ncUVES8IWMFLHkC9n1nPg4Ig5H3wMiZEBjhaWYYBqt2HuXlb3az4qfDnv3DUqK4/dxUxveKx2rVis0i0nYpqIh4i2HAjv/A10/AwY3mvsBIGH0fjLgTAqqvUbXtYD6vfJPBwvT9VLjMr1pqbAgzzu3MlMEdtACiiLRJCioi3uZ2w7ZPzDEsR7ab+0LizPErQ24BR2C15ofyS5m3eg9vrt1LQal5CXR0SAC/HNmJm8/pREyos6X/AhERn1FQEWkpbhdset+8Suj4HnNfeHsYfT/0vKTa1PwAhWWVvLcui1dXZngWQHTarUwZ0oHbxnQmNS60ZesXEfEBBRWRluaqgB/ehBV/hvz9J/fH9YKuF0KXC8zp+R3m/ECVLjeLfzzIyyt2s3FfHmCukzi+Vzx3jE1laKeoGi/fFxFp7RRURHylohTSXjd7WfavB8N98jl7oLmmUNcLzan643pgAN9lHOOf3+zmq605nqYDkiO549xUJvSJx26ztvzfISLiRQoqIv6g+Jh5SfPOr2Dn11CQXf358PZmT0vXCyH1fHYWOHh15W7+nbaf8koz4CRHBzFjdGeuGZpMiFZtFpGzhIKKiL8xDDi8DXYugV1LYM8qcJWdfN5iNRdC7HIhx5POZd7eGN5Ym8Xx4goAIoIcnN8jjoHJkQxMjqR3UjhOu64YEpHWSUFFxN9VlMDeVWZPy64lZog5VWAErpTz+M42iGd2d2Dd8eqXPQfYrPRKCmdQciSDOprhpWN0sMa1iEiroKAi0trk7TvZ27J7GZTmVXu6OLwLh62xHC+DoyVQ5LJSjoNyw045dspxYHcEEB0eRlxkGAnR4STFRBIUFAg2J9gcYHdWvx/eHqI6+ebvFZE2TUFFpDVzVUJ22sng8vNBuc2p9+Vw3m8gvrd3Xl9E5AwUVETOJsXHIOtbKM03x7VUlpmXQ7uqbivLqKws43heIcfyCsgtLCK/sIjy8lICqPRsDkslTksl4XYXoXY30WX7sGBgYMHS9ypzdei4Hr7+a0WkDVBQERGOFZWzISuXH7JySc/KZUNWLnklFZ7nu1uyuM/+by6xmWsXGVg40vkyQi/6LUGJPX1Vtoi0AQoqInIawzDIOFJEelVwSc/KZUt2Pt2MPdxv/zcTbN8D4DIsfO28gA2pd9Cpax8GdYwiNTZECymKSLNRUBGReimtcLF5fx4/ZOZyZMe3nLv/Fca4zcBSaVj5t2ssf3NdQb4ziYEdo6pdZRQZHODj6kWktVJQEZFGO7p9Ne6lc4g7uAKACsPG+67z+HvlFWQT62mXGhfCoOQoBnaMZFByJD0TwjSLrojUi4KKiDRd1nfmYou7vgbAZbGzNuISni2/jHXHgk5rHuSwMTQlilFdYhndNYY+SRHYdLpIRM5AQUVEms/eNbDsj5Bh9rBgC6B0wC9Znzydbw8HmIN1M3MpKKusdlhEkINzUqMZ3TWW0V1jSY0N0YR0IgIoqIiIN+xZCUv/aM6oC+Yii0NvhdH34w5px46cQlbvOsKqnUf5dvfR04JLQnggo7rGMLqLGVwSIgJ98EeIiD9QUBER7zAMs2dl6R8ha625zx4Ew2+D0fdDiDmGpdLlZuP+PFbvNIPL+r3HKXdVn7QuNS7EE1pGpsYQEexo4T9GRHxFQUVEvMswzLErS/8I+82rhHAEQ0xXs6fF7qx2W2kN4HCJhX0FbvbkucgqcFNqOCgjgDIclOMgPjqCLomx9OgQS/f2cTgDgyAwEiKSwa4rjETOJgoqItIyDAN2fgVLn4TsH7zzFlipCEnEEtMZe0xnLNGdISoFoqpug6O98r4i4j0KKiLSsgzDDCrFx6CytGorq+dtKWWlxeQXFlJcVER5WTFWVzlOSwXRFBBsKav1rSsDwjEiU7DHpmKJSoFTg0x4e7DZW+SfQETqryG/3/oGi0jTWSzQfnCjD3cCcVX3DcNg95Eilu48wuZ9eeQe2Y9xLIPgon10shyiozWHjpZDdLLk0M6Si708H3I2mtvPuC123BHJ2GI6Y4nqbIaY9kPMze5sdL0i0nLUoyIirUJJuYvMY8XsOVrE3qNF7DlazMHDRyk/kkFQURYdOUSyJccMM5YcOlgO47RUnvG1Kq1OjkYNoDjxHIyUMYSkjiAmIlwT1om0EJ36EZE2pazSRdaxEk+A2Xu0iL1HCig6kkVAfibtOUQnyyFSLQcYZt1OnCWv+vGGgx+Mrmy09WVn8ECORPYnMiKcuDAn7cICiQtzEhfqNB+HOwlz2jUnjEgTKKiIiFSpcLnZf7ykqiemmJz8Eji6g3ZH15FSmE6f8o3EklvtmDLDTrrRlW/dPVnr7k2auxulnDxV5LRbzfAS5iQ+LJCkyCCSIgPpEBVUdT+ImJAAhRmRGiioiIjUl2HgOrKT4p+W4c5YSeD+NThLDlVrUomdrdZurHH1ZEVFT9a7u1FC7RPWOe1W2leFlhO3SZGBtI8yHydEBOK027z5l4n4LQUVEZHGMgw4ttuciXfvKvM2f3/1JlY7RTH9yYkewu6gfuxwJ7KlJIqs3HKyc0vIKaj9SiUwxx/HhTo9QaZ9VBBJEYF0iApmdNdYggIUYuTspaAiItJcDAOO7zEDy4ktf9/p7ax285Lo2G5URqWSF9SJA44OZBiJ7C4OITuvlOy8EvbnlrD/eAllle7TX6NKalwIL940hO7xYd77u0R8SEFFRMRbDANy98Keqt6Wgxvh6E5zTpiaOMMhpos5c29MN4yYLuQFp7DfmkRWkYX9uaVkVwWY7/ce50hhGUEOG09N6cflA9u33N8m0kIUVEREWpLbbZ4eOrrz5HZkh3mbmwnU8p/ZsMSqAGNueSGd+Muqo6zKqiDfCOGyET35zeSBBNh16bScPRRURET8RUUpHM84JcCcEmaKj9TrJcpxYAuOxBYUAYERZg9NYAQEnriNAGdE9X2ntgkIA6uCjvgPzUwrIuIvHIHQrpe5/VzxMXPg7qk9MMd2QUkulOZDmTnfSwAVUHzY3BpVQwgkDoCkQSe36FSFF2kVFFRERHwlONrcOgw98/NuN/sO5fD7d1ZxKOcQ4ZZibh4UycXdQrCW5ZthpjQXyvKhNK/qcd4pj/PAVQ4VRZC52txOcIab4aX94JPhJbKTeTlSa+GqhMJDUHAA8rOr3xYfg47nwKCbICzB15VKE+jUj4iInyutcPHYxz/yzrosAC7o2Y6/XjuQiGBH3QdXlJpXLWX/cHI7uPHMg3+Doqr3uiQNMhd2bOnwYhhmyCo4CAXZkH/glNsTYeQgFOWAUfPVUwBYbNB9IgyZBl3Hg9VPLvuuKIXsNAiMNBfRDAj2dUUtSmNURETOQu99n8XDH22mrNJNh6ggXpg6hH4dIhr+Qq5KOLzN/KH0hJfN4K44vW1Iu9PDS1h89TaGAa4KqCyBiqqtshQqis0fZM/+qn2Vpae0q7otya3eI1JRXL+/xWqH0ASz1yQ8EcKSzFt7EPy4ALLWnmwb3t7sYRl0E0R2bPi/W1OVFcKOL2HrJ+ZteWH12qJTzYUzo1NPblGdwRna8rV6mYKKiMhZ6sfsPO5+M43MY8UE2K3872V9uG5YctOn668sg5wtJ4PL/h/Mx4br9Lah8WBzVg8dZ2rXVIERJ4OH5zah+r6Q2Np7SXK2QdobsOFtKDlWtdMCXS4we1l6XAy2evRMNVbJcfjpC9jyMexaUr0nK6Sd+e9ellfz8WD+e0d3OXOQCWydv30KKiIiZ7G8kgp+/V46X23NAeCaIR144oq+BDqa+bRGRYnZ03LqaaPD26j1cmss4Ag2BxE7gsEe+LP7Vc/Zg8BxyuYMOyWAVG3NeTqksszsyUh7HTJWnNwfEgcDb4TB08y5bppD0RHY9qkZTjKWg/uUVbyjOkPvy6DXZZA02DytVnLcHFR9bDcc3XXy/rHdp4SrGgTHmoElpirIhLYzT3dZrGaAs1irNssp960n25zpeevPnguOab5/myoKKiIiZzm32+DFFbv4yxfbcRvQKzGcF6YOJiU2xLtvXFYIh7eb9x2BZsg4NXTYAvx/QO7RXfDDv+CH+eY4lxNSzjUDS6/J5t/WEPnZZhDa+om59MKpY2fielWFk8kQ37dh/z4lx+FYRlVwyageYk6t3Zv6Xg1Xv9qsL6mgIiLSRqzedYT/evsHjhSWExZo5/+uGcBFfXSVS724KuCnxeapoR3/wdNTFBQF/a83Tw2d6bLyE45lVIWTj2HfuurPJQ40g0nvyyG2m3fqLyuoHlyO7obio2ZIqu/mdlXdN07Z76repvskmPRUs5auoCIi0oYczCtl5ltprN97HIC7z+/Cr3/RHbtN86TUW94++OFNSPtX9bWcOgw3A0ufKyEgxOxN2vIxbF0IBzdVf43kEeYpnV6TIapTy9bfyiioiIi0MRUuN3M+38ZrqzIAGJkaw3M3DCIuzOnjyloZtwt2fQ3r58H2RScHCTvDzfEfR3eebGuxQcpoM5z0vNQcXyP1oqAiItJGfboxm4c+2EhRuYt2YU7+MXUwQ1OifV1W61RwCNLnm6eGjpsBEKsDuowzw0mPiyEkxrc1tlIKKiIibdjOnALuejONnTmF2K0W7j6/C32SwokLCyQ+3ElcmBOn3U8mPmsN3G5zVt/iY5B6nnnZtDSJgoqISBtXVFbJ7A838fGG7DM+HxnsoF2Yk3ZhgeZt+InbU/c5CQ5ovpVWXG6D4vJKSspdlFS4KC43t9IKF2WVLipcBi63QaXboNLlptJd/XFtz514bL6GG7cBsaFOkqOD6BAVTIeoINpHBjX/JdzSKAoqIiKCYRh8sH4fX209RE5BGTn5ZRwuKKPcVce086cIddqrwsvJABMb5sTlNij9WdgoLq885b6rWiApKXc16H29JS7MSYeok+HlxP3kqCCSvBxkyivdFJRWUFBaSUFpJUXllQQ5bIQHOYgIchAWaMfRRgZAK6iIiMgZGYZBXkkFh/LLyCkoJSe/zAwxBaVVYabUE2pKKrww2yzmNCLBDhtBAXaCAqwEO+w4HVbsVgt2qxW7zYLNasFutWCzWnHU8thus1btP/nYZjXnKTlcUEbWsWL2HS9h3/Fiisrr/nvanRZkTgaa8CBHVcio8NzmV4WOU/cV/GxfftX9ssq6g1pIwMngEh7oIDzIQXiQnfDAqn2e5+zVHwc5CAmwNX2G4haioCIiIk1iGAaFZZWe0HIy1JRypLAcu9VCcICNwAAbwQ77KfdtBAfYCAqwEeSwERxgN+9XPRcUYMNpt7b4D6phGOQWV3hCS/XbErKOF1NcjyDTHEICbIQG2glx2iktd5FXUlGvEFUXm9VCUD17hBry039xv0T+fM2AxpZ1Rg35/W6+k48iInLWsFgshAU6CAt00CWu9S+KZ7FYiAoJICok4IwLOf48yGSdEmJOBJrichdhTjthgfaqfxs7oafcDws0ez5O3A9zOqq1DQ90EBpo9/T4nKrS5Sa/tJL8kgrySyvIK6kgv6TSvC2tIL+kal9p1b4TW1XbE+N7CssqT3vtpqpPT5A3KaiIiEibV58gYxhgPUPIaA52m5XokACiQwIafKxhGJRWuMkvraCk3FXvGfot1K9hUIBvByArqIiIiNTBYrH47RJGFovFc3rtbNQ2hheLiIhIq6SgIiIiIn5LQUVERET8loKKiIiI+C0FFREREfFbCioiIiLit3waVFasWMHkyZNJSkrCYrHw0Ucf+bIcERER8TM+DSpFRUUMGDCA559/3pdliIiIiJ/y6YRvkyZNYtKkSb4sQURERPxYq5qZtqysjLKyMs/j/Px8H1YjIiIi3taqBtPOmTOHiIgIz5acnOzrkkRERMSLWlVQmT17Nnl5eZ4tKyvL1yWJiIiIF7WqUz9OpxOn0+nrMkRERKSFtKoeFREREWlbfNqjUlhYyM6dOz2PMzIySE9PJzo6mo4dO/qwMhEREfEHPg0q33//PePGjfM8fuCBBwCYNm0a8+bN81FVIiIi4i98GlTOP/98DMPwZQkiIiLixzRGRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3FFRERETEbymoiIiIiN9SUBERERG/paAiIiIifktBRURERPyWgoqIiIj4LQUVERER8VsKKiIiIuK3/CKoPP/886SkpBAYGMiIESP47rvvfF2SiIiI+AGfB5V3332XBx54gEcffZS0tDQGDBjAhAkTyMnJ8XVpIiIi4mM+DyrPPPMMt99+O7fccgu9e/fmxRdfJDg4mNdee83XpYmIiIiP2X355uXl5axfv57Zs2d79lmtVsaPH8+aNWtOa19WVkZZWZnncV5eHgD5+fneL1ZERESaxYnfbcMw6mzr06By5MgRXC4X8fHx1fbHx8ezbdu209rPmTOHxx9//LT9ycnJXqtRREREvKOgoICIiIha2/g0qDTU7NmzeeCBBzyP3W43x44dIyYmBovF0qzvlZ+fT3JyMllZWYSHhzfra0vj6XPxX/ps/JM+F//Vlj8bwzAoKCggKSmpzrY+DSqxsbHYbDYOHTpUbf+hQ4dISEg4rb3T6cTpdFbbFxkZ6c0SCQ8Pb3P/A2oN9Ln4L302/kmfi/9qq59NXT0pJ/h0MG1AQABDhgxhyZIlnn1ut5slS5YwcuRIH1YmIiIi/sDnp34eeOABpk2bxtChQxk+fDjPPvssRUVF3HLLLb4uTURERHzM50Hluuuu4/DhwzzyyCMcPHiQgQMHsnjx4tMG2LY0p9PJo48+etqpJvEtfS7+S5+Nf9Ln4r/02dSPxajPtUEiIiIiPuDzCd9EREREaqKgIiIiIn5LQUVERET8loKKiIiI+C0FlTN4/vnnSUlJITAwkBEjRvDdd9/5uqQ277HHHsNisVTbevbs6euy2pwVK1YwefJkkpKSsFgsfPTRR9WeNwyDRx55hMTERIKCghg/fjw7duzwTbFtTF2fzfTp00/7Dk2cONE3xbYhc+bMYdiwYYSFhdGuXTuuuOIKtm/fXq1NaWkpM2fOJCYmhtDQUKZMmXLaRKhtmYLKz7z77rs88MADPProo6SlpTFgwAAmTJhATk6Or0tr8/r06cOBAwc828qVK31dUptTVFTEgAEDeP7558/4/J/+9Ceee+45XnzxRb799ltCQkKYMGECpaWlLVxp21PXZwMwceLEat+ht99+uwUrbJuWL1/OzJkzWbt2Lf/5z3+oqKjgoosuoqioyNPmV7/6FZ988gnvv/8+y5cvJzs7m6uuusqHVfsZQ6oZPny4MXPmTM9jl8tlJCUlGXPmzPFhVfLoo48aAwYM8HUZcgrAWLBggeex2+02EhISjD//+c+efbm5uYbT6TTefvttH1TYdv38szEMw5g2bZpx+eWX+6QeOSknJ8cAjOXLlxuGYX5HHA6H8f7773vabN261QCMNWvW+KpMv6IelVOUl5ezfv16xo8f79lntVoZP348a9as8WFlArBjxw6SkpJITU1l6tSpZGZm+rokOUVGRgYHDx6s9v2JiIhgxIgR+v74iWXLltGuXTt69OjB3XffzdGjR31dUpuTl5cHQHR0NADr16+noqKi2vemZ8+edOzYUd+bKgoqpzhy5Agul+u0WXHj4+M5ePCgj6oSgBEjRjBv3jwWL17MCy+8QEZGBueeey4FBQW+Lk2qnPiO6PvjnyZOnMgbb7zBkiVLePrpp1m+fDmTJk3C5XL5urQ2w+12c//99zN69Gj69u0LmN+bgICA0xbY1ffmJJ9PoS9SH5MmTfLc79+/PyNGjKBTp0689957zJgxw4eVibQO119/ved+v3796N+/P126dGHZsmVceOGFPqys7Zg5cyabN2/W+LoGUo/KKWJjY7HZbKeNtj506BAJCQk+qkrOJDIyku7du7Nz505flyJVTnxH9P1pHVJTU4mNjdV3qIXMmjWLTz/9lKVLl9KhQwfP/oSEBMrLy8nNza3WXt+bkxRUThEQEMCQIUNYsmSJZ5/b7WbJkiWMHDnSh5XJzxUWFrJr1y4SExN9XYpU6dy5MwkJCdW+P/n5+Xz77bf6/vihffv2cfToUX2HvMwwDGbNmsWCBQv4+uuv6dy5c7XnhwwZgsPhqPa92b59O5mZmfreVNGpn5954IEHmDZtGkOHDmX48OE8++yzFBUVccstt/i6tDbtwQcfZPLkyXTq1Ins7GweffRRbDYbN9xwg69La1MKCwur/T/wjIwM0tPTiY6OpmPHjtx///384Q9/oFu3bnTu3JmHH36YpKQkrrjiCt8V3UbU9tlER0fz+OOPM2XKFBISEti1axf/8z//Q9euXZkwYYIPqz77zZw5k7feeouFCxcSFhbmGXcSERFBUFAQERERzJgxgwceeIDo6GjCw8O59957GTlyJOecc46Pq/cTvr7syB/97W9/Mzp27GgEBAQYw4cPN9auXevrktq86667zkhMTDQCAgKM9u3bG9ddd52xc+dOX5fV5ixdutQATtumTZtmGIZ5ifLDDz9sxMfHG06n07jwwguN7du3+7boNqK2z6a4uNi46KKLjLi4OMPhcBidOnUybr/9duPgwYO+Lvusd6bPBDDmzp3raVNSUmLcc889RlRUlBEcHGxceeWVxoEDB3xXtJ+xGIZhtHw8EhEREambxqiIiIiI31JQEREREb+loCIiIiJ+S0FFRERE/JaCioiIiPgtBRURERHxWwoqIiIi4rcUVERERMRvKaiISKtjsVj46KOPfF2GiLQABRURqbfp06djsVhO2yZOnOjr0hpk3bp1JCUlAZCdnU1QUBDl5eU+rkpEzkSLEopIg0ycOJG5c+dW2+d0On1UTeOsWbOG0aNHA/DNN98wdOhQAgICfFyViJyJelREpEGcTicJCQnVtqioKM/zFouFF154gUmTJhEUFERqaioffPBBtdfYtGkTF1xwAUFBQcTExHDHHXdQWFhYrc1rr71Gnz59cDqdJCYmMmvWrGrPHzlyhCuvvJLg4GC6devGxx9/XO+/YfXq1Z6gsnLlSs99EfE/Cioi0uwefvhhpkyZwoYNG5g6dSrXX389W7duBaCoqIgJEyYQFRXFunXreP/99/nqq6+qBZEXXniBmTNncscdd7Bp0yY+/vhjunbtWu09Hn/8ca699lo2btzIxRdfzNSpUzl27FiNNa1cuZLIyEgiIyP54IMP+N3vfkdkZCQvvvgizz33HJGRkTz11FPe+QcRkcbz9fLNItJ6TJs2zbDZbEZISEi17cknn/S0AYy77rqr2nEjRoww7r77bsMwDOPll182oqKijMLCQs/zn332mWG1Wo2DBw8ahmEYSUlJxu9+97sa6wCM3//+957HhYWFBmAsWrSoxmNKSkqMjIwMY9GiRUZUVJSxe/du4/vvvzcCAgKMrVu3GhkZGcbx48cb9O8hIt6nMSoi0iDjxo3jhRdeqLYvOjq62uORI0ee9jg9PR2ArVu3MmDAAEJCQjzPjx49Grfbzfbt27FYLGRnZ3PhhRfWWkf//v0990NCQggPDycnJ6fG9oGBgaSkpPDee+8xadIkOnfuzOrVqzn33HPp2bNnre8lIr6joCIiDRISEnLaaZjmFBQUVK92Doej2mOLxYLb7a6xfWhoKABlZWVYrVYWLlxIeXk5hmEQGhrKueeey6JFixpfuIh4hcaoiEizW7t27WmPe/XqBUCvXr3YsGEDRUVFnudXrVqF1WqlR48ehIWFkZKSwpIlS5q1pvT0dL7//ntsNhtLliwhPT2dmJgY3nvvPdLT03nllVea9f1EpHmoR0VEGqSsrIyDBw9W22e324mNjfU8fv/99xk6dChjxoxh/vz5fPfdd7z66qsATJ06lUcffZRp06bx2GOPcfjwYe69915uvvlm4uPjAXjssce46667aNeuHZMmTaKgoIBVq1Zx7733Nrrurl27snbtWuLj4xkzZgyZmZkUFBQwefJk7Hb9p1DEX+nbKSINsnjxYhITE6vt69GjB9u2bfM8fvzxx3nnnXe45557SExM5O2336Z3794ABAcH88UXX3DfffcxbNgwgoODmTJlCs8884zn+GnTplFaWspf//pXHnzwQWJjY7n66qubXPuyZcsYO3YsAMuXL2fkyJEKKSJ+zmIYhuHrIkTk7GGxWFiwYAFXXHGFr0sRkbOAxqiIiIiI31JQEREREb+lk7Mi0qx0NllEmpN6VERERMRvKaiIiIiI31JQEREREb+loCIiIiJ+S0FFRERE/JaCioiIiPgtBRURERHxWwoqIiIi4rf+P7DUTIYs7LGYAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(history.history['loss'], label='loss')\n",
"plt.plot(history.history['val_loss'], label='val_loss')\n",
"plt.ylim([0, max(plt.ylim())])\n",
"plt.xlabel('Epoch #')\n",
"plt.ylabel('CE/token')\n",
"plt.legend()"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:10.031621Z",
"iopub.status.busy": "2023-12-07T12:24:10.031364Z",
"iopub.status.idle": "2023-12-07T12:24:10.186521Z",
"shell.execute_reply": "2023-12-07T12:24:10.185872Z"
},
"id": "KkhXRASNG80_"
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABgb0lEQVR4nO3dd3hUVf7H8ffMpPeEkEog9N6kxIAUAQVUVqzAoiAqdn66WVbFAnZsKBZWFuwVlNUVRbFEijRRkCLSW2hpQHqfub8/JgzEBEggySSTz+t57pOZO+fOfCeF+XDuueeYDMMwEBEREXERZmcXICIiIlKdFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FDdnF1DbbDYbhw8fxt/fH5PJ5OxyREREpBIMwyA7O5uoqCjM5jP3zTS4cHP48GFiYmKcXYaIiIicgwMHDtCkSZMztmlw4cbf3x+wf3MCAgKcXI2IiIhURlZWFjExMY7P8TNpcOHmxKmogIAAhRsREZF6pjJDSjSgWERERFyKwo2IiIi4FIUbERERcSkNbsxNZVmtVoqLi51dhtRh7u7uWCwWZ5chIiJ/oXDzF4ZhkJycTEZGhrNLkXogKCiIiIgIzZkkIlKHKNz8xYlgExYWho+Pjz60pEKGYZCXl0dqaioAkZGRTq5IREROULg5hdVqdQSbRo0aObscqeO8vb0BSE1NJSwsTKeoRETqCIWbU5wYY+Pj4+PkSqS+OPG7UlxcrHAjIg2azWaQW1RCVkEJNptBTIjzPksVbiqgU1FSWfpdEZFTZeYVsy05i61HstiWnE1KVgE+nm74ebjh5+WGr6cbfp6W0q/2zfcvX/083fByN9f6vy82m0F2YQlZ+cVkFRSTlV9S+rWY7IKScvvKtSkswTDszxXXPIT5t8fXav2nUrgRERGpohKrjX1Hc9l6JJttyVlsO5LN1iNZHM4sqJbnt5hN+HhY8C8NPScDkAWzyYTNMLAZ9kBiMwyshn0soLX0vs1GaZu/PnbqMQaGAUUlNrIKisk5JZycDw+LGYvZuf/xU7gREZEal1dUAoCXmwWzkz/4qup4bhFbTwkw25Kz2ZGSTWGJrcL20UHetI8MoH2kP9FB3uQXW8ktLCGn8MRX+5Zbup28byW3yB4wrDaD7IISsgtKavndgpe7mQAvdwK83fH3cnPcDvByK/3qToB32f3+p+zzcnf+KXqFGxEROW9FJTYOZeRz4FgeB47nceBYPgeO53HwWB5Jx/I4nndy3jBPNzPeHhZ83C14eVjwdi/dPE7z9cTtU+57eVjwcrNgMZuwmMFsMmE2mbCYT349sb/svlNum0yYzTj2Aew/mld6WinbcXopJauwwvfs42GhbYQ/7SIC6BDpT7vIANpG+BPg5X7O30ebzSC/2FomAOUUlpBTUEJukT0gYRiYS2s2m8q+d5Pp5Ps59fHTPmY24W4xO4KLv5cbnm7ODyfnS+FGRETOymYzSM0u5MDxPJKOlg8wR7IKKn1Ko7DERmGJjQzqz0SpTUN8aB9pDzInvjYN8an2Xiiz2eQ4DRVerc/csCjcSI0pLi7G3f3c/wcjIs6xLz2X7/9MJulYHknH8jl4LI+DGfkUneY0zAle7mZign2ICfEhJtjb/jXEh5hgH5qEeONmNpFfZCW/2EpBsZW8Imv5+8X2fafeLyg+2S6v9DH7fhs2m33siNVmOE7nWA2jzH6bYx+Ofafj5+lGuwh/2jmCjL03xs+zih+Xx/fD6tchdSvE9IaWg6BJb3DzqNrz1BSbDVK3wN7lkHUYQttARCdo3B486v8Vwwo3Z2EY9i5CZ/B2t1RptPzixYt56qmn+OOPP7BYLMTHx/PKK6/QsmVLAA4ePMi//vUvvvvuOwoLC2nfvj2zZs0iLi4OgK+++oonnniCzZs34+fnR79+/fjiiy8A+1VBX3zxBSNHjnS8XlBQEDNnzuSmm25i3759NG/enHnz5vHvf/+bX375hdmzZzNixAjuueceli9fzvHjx2nZsiUPPfQQY8aMcTyPzWbjxRdfZM6cORw4cIDw8HBuv/12Hn74YQYNGkSHDh14/fXXHe3T0tKIjo7m22+/ZfDgwefzLRaRUxRbbcxZvodXEndWGGQsZhNRQV72ABPsQ0xI2QAT6udx1n+zfDzqxseOI/D8JfQEeLmd31VKqVthxcuweQEYpZ8d+36Gn2eAuy/E9rUHnRYXQ+O2UFtXRBkGHNsDe5fBnmX2mvKOVtDQBI1aQnhHCO9c+rUjBDWtvVqrQd34LavD8outdJj6nVNe+88nhlbpH4Lc3FwSEhLo0qULOTk5TJ06lauuuooNGzaQl5fHgAEDiI6OZuHChURERLB+/XpsNvs/YIsWLeKqq67i4Ycf5v3336eoqIhvvvmmyjU/+OCDzJgxg+7du+Pl5UVBQQE9evTggQceICAggEWLFnHjjTfSsmVLevfuDcCUKVOYO3cuL7/8MhdddBFHjhxh27ZtANx6663cc889zJgxA09PTwA+/PBDoqOjGTRoUJXrE5GK/XEokwf+u4kth7MA+6W8vWJD7AGmtDcmMtALN4trrLdsNpswU40f1gd+hRUvwfZT/t1scTG0uxwOrIU9SyA3DXZ+b98A/KOg5cX2di0Ggl/j6qsH7D0ye5fbw8ze5ZB1sOzj7r7QrA+EtID07ZD8B+Slw9Fd9u3PL0+29QyAsA723p3wjhDeCcLag6d/9dZcTUyGUR0XftUfWVlZBAYGkpmZSUBAQJnHCgoK2Lt3L82bN8fLywuwj/CvL+Hmr9LT02ncuDGbN29m1apVTJ48mX379hESElKubZ8+fWjRogUffvhhhc9V2Z6bmTNncu+9956xriuuuIJ27drx4osvkp2dTePGjXn99de59dZby7UtKCggKiqK2bNnc/311wPQtWtXrr76aqZNm1aF70bNqOh3RqQ+KSi28mriTv6zfA9Wm0GQjzvTRnRgZLdozeN0NoYBu3+y99Ts+7l0pwnaj4B+CRDV/WTbE6eBdi+xB539q6DkL5eNh3e2h52WF0PTeHD3rlo9ecfsdZwINEd3ln3c7G4/RdZ8ADTvD9E9yp8my06BlD8gZcvJLW0b2E4zPiq4+cmwE97RHn6CYsFc/SH4TJ/ff6Wem7Pwdrfw5xNDnfbaVbFz506mTp3KL7/8Qnp6uqNXJikpiQ0bNtC9e/cKgw3Ahg0bmDhx4nnX3LNnzzL3rVYrzzzzDJ9++imHDh2iqKiIwsJCx8y+W7dupbCw8LSnl7y8vLjxxht5++23uf7661m/fj1//PEHCxcuPO9aRRq63/Yd4/7/bmJPWi4Al3eJ5PG/dSTUz9PJlZ3CMKAoF/KP2U+j5B2D/OOlp1RM0PRC+wdrDXyYnpbNClu/soeaIxvs+8xu0GU09L0XGrcpf4zZDBGd7Vvf/4PiAkhabQ86u3+C5M2QUrqtehXcvOwB50TPTkXvsTDH/hwnTjUlbwZO7a8wQVS3k2GmafzZx9P4h9u3Vqf8m1xSZA9KKVtOBp/kPyAnGY7vtW/bvj7Z3t3X/np/n1f572k1U7g5C5PJVGfOEZ/NiBEjaNasGXPnziUqKgqbzUanTp0oKipyrIN0Omd73GQy8ddOvhPLVZzK19e3zP0XXniBV155hZkzZ9K5c2d8fX257777KCoqqtTrgv3UVLdu3Th48CDvvPMOgwYNolmzZmc9TkQqllNYwguLt/H+mv0YBjT29+SpkZ0Y2jGiZl/YMKAgszSonNiOnnL/aNnHTgQaa9GZn9c7BJr3s3+ItxhoP81SE71OJUWwaT6snGk/bQPg7gMXjIc+90Bgk8o/l7vXyV6aS56AnDR7SNn9k713J/uwPfjsWWJv79vY/t5i+0HWIXuYOfQb2P4yD07jdifDTGxf8A4+//ft5nFy7A3Xn9yfm162hydlM6Rug+Lc8r1Stax+fGrLWR09epTt27czd+5c+vXrB8CKFSscj3fp0oU333yTY8eOVdh706VLFxITE5kwYUKFz9+4cWOOHDniuL9z507y8vLOWtfKlSu58sorueGGGwD74OEdO3bQoUMHAFq3bo23tzeJiYkVnpYC6Ny5Mz179mTu3Ll8/PHHZQYXi0jVLNuRxkOfb+ZQRj4Ao3rG8NBl7Qn0qeYrG3PS4PDvJ7fkTZCdfHKQbVVZPMCnkT3I+ITYP7SL82D/ansI+vPLk2NEAprYP9xblH7IB0Sd33spyoV179mvfso6ZN/nFQRxt0Pv28G3GhZa9msMna+1b4YB6TtOBp19K+zjdTZ/Zt9OFdS0NMyUvlf/WryA3DfU/j1uMeDkPmsJHNsNJRXPDVRbFG5cRHBwMI0aNWLOnDlERkaSlJTEgw8+6Hh8zJgxPPPMM4wcOZLp06cTGRnJ77//TlRUFPHx8UybNo3BgwfTsmVLRo8eTUlJCd988w0PPPAAAIMGDeL1118nPj4eq9XKAw88UKnLvFu3bs2CBQtYtWoVwcHBvPTSS6SkpDjCjZeXFw888AD3338/Hh4e9O3bl7S0NLZs2cItt9zieJ4TA4t9fX256qqrqvm7J+L6MvKKePLrrfx3vX1QaZNgb569ugsXtQ49/yfPO1Y2yBzeUH7w6qncfe1BxSe4NKw0Kg0spcHFp5E9vDhuh4CHb8W9MdZiOLTu5KDZg2vtr73xY/sG0Kh1adAZALEX2Z+3su9r7Rz4Zbb9VBiAX4S9l6bHTTU3mNZksl9J1bgtXHinvcfo4K/2sJO0xh5gToSZkOY1U8O5srjZ63YyhRsXYTabmTdvHv/3f/9Hp06daNu2La+++ioDBw4EwMPDg++//55//vOfXHbZZZSUlNChQwdmzZoFwMCBA/nss8948sknefbZZwkICKB///6O558xYwYTJkygX79+REVF8corr7Bu3bqz1vXII4+wZ88ehg4dio+PD7fddhsjR44kMzPT0ebRRx/Fzc2NqVOncvjwYSIjI7njjjvKPM+YMWO47777GDNmjAbuilTRt5uP8OiXW0jPKcRkggl9mjN5aJtzO+WenwFHNpYNMxn7K2hoss+dEtW9dOsGQc3swcKtGsf0WNzt426aXggDH4CivNJxKMvtp3kOb7CPFzm6E359015XZJfSU1gDSsehlD2dTtZhWD0LfnvHfooF7Ke6+t4LXcdUb/2V4eZhP8UU27d2X7ce09VSp9CVL3XXvn37aNmyJb/++isXXHCBs8tx0O+M1GWp2QVM+3IL3/6RDECrMD+eu6YLPZpVchxGYTYc2VQ2yBzbXXHbkJanBJnu9gBRFy4Tzj8O+1aeHHSbvr3s42Z3aNLLHnSie8LWL2HjvJPjfMI7Q79/QIeRYK7/yxLUZ7paSlxGcXExR48e5ZFHHuHCCy+sU8FGxKnyjsGuH+3zquz6CYqywWQBswXDZKHEMOFVYvC0YeJJTzNeHh74Gh6YPnezn/YwWxzt7V/NYDKf3FeQCek7KXv1TamgZn8JMl3BO6i2vwOV4x0M7a+wb2Af9+OY+2UZZB6ApFX27VRN+9gv5241pF5NXid2CjdSp61cuZKLL76YNm3asGDBAmeXI+JcR3fD9m/tW9Lq8oNzDRvYijEB7qWbY5664tKtqgKa2E8pnRpmKjtmpS7yj4Au19s3w7BfxnxivM6h3yCsI1x0n/00l9RbCjdSpw0cOLDcJegiDYbNap/ddkdpoEnfUfbxsI7Qdji0HY7NP5oFv+3nP8t2UVhcjLcFbunblGu7R+JmMkqDj9UeiGy20q+l9x2PndLGzRMiuoBfmHPee20wmexjaUJaQM+KrxSV+knhRkSkLinMsV8Vs/1b2Pld2fV/zG72q33aDIe2wyA4lqISG7vTcnj04z/4bf9xIJjesSE8e01nWjT2c9rbEHEmhRsREScottrILighp6CE/KNJeOz+Dv/9PxCcugbLKVPd51v8+dP3Qn7zjGONpTtpmZ5k/1xCzg87yS7cWmaBS18PCw9e1p6xvZtiNmuciDRcCjciIueh2GojM7+YjLwijucVczy3iIy8Yo7nFZFxYn9u6f3S/Zn5RbSy7uESyzqGmNfTybyvzHPus4Xzg60HibYL+M3WhpLcE/9UF5ZuZZlMMLhdGI9f2YnooCquRyTighRuRET+wmozOJyRz76juew7mkd6diEZpWHleN6JIFNERm4x2YUnp7/3opAQsgk25RBkyiaEbIJMObQhh2BTNsGl+1pbDhHpdsxxnA0TWyztWO91IVv8+pLl2xw/b3fae7nR29MNPy83/L3c8Su9HeDlhp+nO35ebvZ9nm5Y1FMj4qBwIyINks1mcDgzn33peew9msv+9Fz2Hc1lb3ouB47lU2S1YcJGa9MhIk3HCCabEFM2LU3ZBJNDsCmHYLIJ9igNMqZsvKpwOZLh7ovR4mLM7S7D3PpSOvs1pnMNvl+RhqROhJtZs2bxwgsvkJycTNeuXXnttdfo3bt3hW0HDhzIsmXLyu2/7LLLWLRoUU2XKiL1iM1mcCSrgP3puew9msu+9Fz2puex72guScfyyoxXOaExGVxh3sQAjz/ob9lMsJFZwTOfgdm97LIBPsFl10TyaQQBUZhiLsTkrokfRWqC08PN/PnzSUhIYPbs2cTFxTFz5kyGDh3K9u3bCQsrfwni559/7lhRGuwLRnbt2pXrrruuNst2ObGxsdx3333cd999zi5F5Jwczy1i9Z6jbDyQwd7SXpj9R/MorCDAnOBuMdEi2I1LfPfQh020y/2VkJxTLrc2sK+D1KhFBWsg/XU9pNLbnv6a9E3EyZwebl566SUmTpzoWI169uzZLFq0iLfffrvMwo8n/HVF63nz5uHj43PacFNYWEhh4ckBeFlZWdVYvYg4S0GxlXX7j7NiVzordqbzx+FMKpoSyc1sIibEh9hGPsSG+tK8kQ8d3I/QMmstgYeXY96/CnLyyx4U2RVaDoaWgyAmzr62j4jUG04NN0VFRaxbt44pU6Y49pnNZoYMGcLq1asr9RxvvfUWo0ePxtfXt8LHp0+fzuOPP14t9UrdZLVaMZlMmM1mZ5ciVWGzwq5EKMkH3zDwbQx+jcEzoMKeD5vN4M8jWY4w8+u+Y+V6ZVqH+RHXIoRWjf3sQSbUl+ggb9wKM2DPUtidCGuWQNahsk/uF2EPMi0HQcuLwbcaVsoWEadx6qdBeno6VquV8PDwMvvDw8NJTk4+6/Fr167ljz/+4NZbbz1tmylTppCZmenYDhw4ULUiDQOKcp2zVXJm3jlz5hAVFYXNVvYf+iuvvJKbb76Z3bt3c+WVVxIeHo6fnx+9evXixx9/rNr34RQvvfQSnTt3xtfXl5iYGO666y5ycnLKtFm5ciUDBw7Ex8eH4OBghg4dyvHjxwGw2Ww8//zztGrVCk9PT5o2bcrTTz8NwNKlSzGZTGRkZDiea8OGDZhMJvbt2wfAu+++S1BQEAsXLqRDhw54enqSlJTEr7/+yiWXXEJoaCiBgYEMGDCA9evXl6krIyOD22+/nfDwcLy8vOjUqRNff/01ubm5BAQElFvi4X//+x++vr5kZ2ef8/dL/sJmgy1fwL8vhI+vg0/HwTvD4PUe8GxTeCocXuoI/xlA/rtXs3vuOL5/5XZefvL/eHPWs6z47jOO7l5HQMkxIv3cuLp7NC9d35VfHhrMDwkDeGpkZ266sAkDvXbTbOPLuL09BJ5vAQsmwO8f2oONxRNaXAyXPAl3roJ/boOr3oAu1ynYiLgAp5+WOh9vvfUWnTt3Pu3gYwBPT088Pc9jefriPHgm6tyPPx8PHQaPinukTnXdddcxadIklixZwuDBgwE4duwYixcv5ptvviEnJ4fLLruMp59+Gk9PT95//31GjBjB9u3badq0aZXLMpvNvPrqqzRv3pw9e/Zw1113cf/99/Pvf/8bsIeRwYMHc/PNN/PKK6/g5ubGkiVLsFrt6+BMmTKFuXPn8vLLL3PRRRdx5MgRtm3bVqUa8vLyeO6553jzzTdp1KgRYWFh7Nmzh/Hjx/Paa69hGAYzZszgsssuY+fOnfj7+2Oz2Rg+fDjZ2dl8+OGHtGzZkj///BOLxYKvry+jR4/mnXfe4dprr3W8zon7/v51YHXj+s4wYOf38NOTkLzZvs87GELbQE4q5KbbF3+0FkLWQcg6iDfQsnS7FOAvZ4eMEhOm/SGQ3hg2Nbb3/pQUwL4VUPiXU9CN20OrwfaemaZ9wMOnxt+yiDiHU8NNaGgoFouFlJSUMvtTUlKIiIg447G5ubnMmzePJ554oiZLrBeCg4MZPnw4H3/8sSPcLFiwgNDQUC6++GLMZjNdu3Z1tH/yySf54osvWLhwIffcc0+VX+/UQcexsbE89dRT3HHHHY5w8/zzz9OzZ0/HfYCOHTsCkJ2dzSuvvMLrr7/O+PHjAWjZsiUXXXRRlWooLi7m3//+d5n3NWjQoDJt5syZQ1BQEMuWLeOKK67gxx9/ZO3atWzdupU2bdoA0KJFC0f7W2+9lT59+nDkyBEiIyNJTU3lm2++Oa9eLim1Zxn89BQcXGu/7+EP8XdT0PMOfku2smJXOit3pbPzcBqNjExCTZk0MmURbs6iU1ARHQIKaOqZRzCZmHPTIDcNctMxYdiXJ8g7Cml/CcjeIfYgc+J0U4CT/pMiIrXOqeHGw8ODHj16kJiYyMiRIwH7KYvExMSzfuh+9tlnFBYWcsMNN9Rske4+9h4UZ3Cv/P8sx44dy8SJE/n3v/+Np6cnH330EaNHj8ZsNpOTk8Njjz3GokWLOHLkCCUlJeTn55OUlHROZf34449Mnz6dbdu2kZWVRUlJCQUFBeTl5eHj48OGDRtOO8B769atFBYWOkLYufLw8KBLly5l9qWkpPDII4+wdOlSUlNTsVqt5OXlOd7nhg0baNKkiSPY/FXv3r3p2LEj7733Hg8++CAffvghzZo1o3///udVa4N2YK29p2bvcgBKLF6sD7+WT9yvZuN6C/t/WI3VdurpVw98w5tzQatQLmoVSlyLRvh5nuafKZsV8o5Bbqo97OSUhh7DCs362gcFmy01/x5FpM5x+mmphIQExo8fT8+ePenduzczZ84kNzfXcfXUuHHjiI6OZvr06WWOe+uttxg5ciSNGjWq2QJNpkqdGnK2ESNGYBgGixYtolevXvz888+8/PLLAEyePJkffviBF198kVatWuHt7c21115b5pL6ytq3bx9XXHEFd955J08//TQhISGsWLGCW265haKiInx8fPD2Pv3072d6DHAMCj51JfDi4vITo3l7e2P6y6DT8ePHc/ToUV555RWaNWuGp6cn8fHxjvd5ttcGe+/NrFmzePDBB3nnnXeYMGFCudeR08vMK2ZXWjbpO3+jxeaXaZ25CoAiw8LH1sHMKriStD3BQIHjmPAAT/q2CqVf61D6tgwlLKCSc7+YLfYByH6Na+CdiEh95vRwM2rUKNLS0pg6dSrJycl069aNxYsXOwYZJyUllbsKZvv27axYsYLvv//eGSXXSV5eXlx99dV89NFH7Nq1i7Zt23LBBRcA9sG9N910E1dddRUAOTk5jsG5VbVu3TpsNhszZsxw/Fw+/fTTMm26dOlCYmJihVeptW7dGm9vbxITEyscCN64sf2D6siRIwQHBwP2HpfKWLlyJf/+97+57LLLADhw4ADp6ell6jp48CA7duw4be/NDTfcwP3338+rr77Kn3/+6Th1JicZhkFaTiG7UnLYlZbDrtQcdpbeDsjZwz/cFnCF5RcASgwz/7X259WSqyjwjaZljB+XhPnROsyPVqVbRICXAqSIVCunhxuAe+6557SnoZYuXVpuX9u2bcv8z17sxo4dyxVXXMGWLVvKnK5r3bo1n3/+OSNGjMBkMvHoo4+Wu7Kqslq1akVxcTGvvfYaI0aMYOXKlcyePbtMmylTptC5c2fuuusu7rjjDjw8PFiyZAnXXXcdoaGhPPDAA9x///14eHjQt29f0tLS2LJlC7fccgutWrUiJiaGxx57jKeffpodO3YwY8aMStXWunVrPvjgA3r27ElWVhb/+te/yvTWDBgwgP79+3PNNdfw0ksv0apVK7Zt24bJZGLYsGGAffzS1Vdfzb/+9S8uvfRSmjRpck7fJ1dRYrWxKy2HTQcz2Xwwkz+PZLEzJZusgpIy7WJMKTzo9jkjPVZgMRn2tZKCh7C38ySaN+/MV2F+hPhqrhgRqR11ItxI9Rg0aBAhISFs376dv//97479L730EjfffDN9+vRxhItzncywa9euvPTSSzz33HNMmTKF/v37M336dMaNG+do06ZNG77//nseeughevfujbe3N3FxcYwZMwaARx99FDc3N6ZOncrhw4eJjIzkjjvuAMDd3Z1PPvmEO++8ky5dutCrVy+eeuqpSs1A/dZbb3HbbbdxwQUXEBMTwzPPPMPkyZPLtPnvf//L5MmTGTNmDLm5ubRq1Ypnn322TJtbbrmFjz/+mJtvvvmcvkdOl3sUtnwOf35pP3XTqFXp1hoatYSgphWORbHaDPam24PMpoOZbD6UyZbDmRQUlw/CZhM0DfGhZ0gBfy+cT7e0rzAbpYGn7eWYBz1M5/COWitJRJzCZDSwLpCsrCwCAwPJzMwkICCgzGMFBQXs3buX5s2b4+WlNV8aqg8++IB//OMfHD58GA+PM/c21JnfmZJC2PEdbJpv/2o7wwKOFg+MkBbk+cdy2NKE7cVh/JLdiCVpARws8gHKniLy83SjU3QAXZoE0TEqgDbh/jT3zsNrzavw65v2S7fBfkXSoEcgukfNvU8RabDO9Pn9V+q5ESmVl5fHkSNHePbZZ7n99tvPGmyczjDg4G+w8RP4479QkHHysciu0GU0eAVgpO8iP3kb1rRdeGfvw81ahCltG75p22gNtAauADBDpqcvKR5NyPdvjltYGxo160BYs46YQ1vZ54XJz4BVr8CaN6A41/5aTeNh0KMQ27e2vwMiIhVSuJEyPvroI26//fYKH2vWrBlbtmyp5Ypqz/PPP8/TTz9N//79yywJUucc3w+bPrWHmmO7T+73j4Iu10PX0aR6NWf+rwf4betxNh8K51huLwDM2IgypdPCdITWlmQu8D1KW/cUIksO4pt/hEBTLoHF2+HYdji2GE6dOiagCRRmQ2HpKtmR3eyhptVgLRQpInWKTkudos6cYnCi7OzscpMqnuDu7k6zZs1quaK6rdZ+Zwoy7WNoNs6D/StP7nf3gfZ/g66joXl/dqTlMXf5Hv634RDF1pN/2m5mE+0i/ekcHUSXJoF0jg6kbYQ/7pZTrkQsyoNje+DorrJb+s6yvUKN28Ogh6HdFQo1IlJrdFrqPDWwvFeGv7+/lhqoghr9XbGWwJ4l9h6abYvsywoAYILm/aHrGGg/AsPDl1W7jzL3vXUs3Z7mOLxHs2BGdouiS5Mg2kb44+V+lgntPHwgopN9O5Vh2CfLO7oLrEXQrI8mxxOROk3h5hTu7u6AfexFZSZ8E8nLywNO/u5Ui+TN9h6aTZ/aZ989IbStvYemy/UQ2IRiq41Fm44wZ/nv/HnEfvWbyQTDOkZwa78W9GgWXD31mEzg28i+iYjUAwo3p7BYLAQFBZGaav9A8fHx0eRiUiHDMMjLyyM1NZWgoCAslvPsycg7Bhs+soealD9O7vdpBJ2utYeaqO5gMpFdUMy85Xt4e+VejmTae3O83S1c17MJt1zUnGaN6v6M2iIiNUnh5i9OLNh5IuCInElQUNBZF3k9I2sx/PoWLH3GPq4GwOIBbYfbTzu1GgIWe6/Q4Yx83l21j09+SSK70D6nTKifJzf1acbYuGYEa5I8ERFA4aYck8lEZGQkYWFhFa5pJHKCu7v7+fXY7P4JFk85uZp1WEfofSt0vAq8T55S2nI4kzd/3stXGw9TUrrIZKswPyb2a86V3aLPPpZGRKSBUbg5DYvFcv6nGkQqcnQ3fP8IbP/Gft+nkf2S6gvGOQbqGobBsh1pzP15Dyt3HXUcGt+iERP7N2dgmzDMZp0yFRGpiMKNSG0pzIblL8Kaf9uvOjK7Qe/bYMD9jp6awhIrCzcc5s2f97I9JRsAi9nE5Z0jmdivBZ2bBDrzHYiI1AsKNyI1zWaDTfPgx8cgp3QOoZaDYNiz0LgtABl5RXy8Nol3V+4jNdu+nIGvh4XRvZsyoW8sTYJ9nFS8iEj9o3AjUpMO/AqLH4BD6+z3Q1rA0OnQZiiYTGw9ksV7q/bxvw2HHAtURgR4MaFvLKN7NyXQuxovMRcRaSAUbkRqQtYRe0/Npnn2+x7+MOBfEHcHJSZ3vv8jmXdX7WPt3mOOQ9pHBjCxX3Ou6BKFh5u54ucVEZGzUrgRqU7FBbBmFiyfcXJhyW43wOCppJuCmLc8iY9+SXLMT2MxmxjWKYKb+sTSs1mw5lUSEakGCjci1cEw7EskfP8wHN9n39ekNwx/lo22lrz37T6+3rieIqv91FOonwdjejdlbFwzIgIb5jpmIiI1ReFG5Hyl/AmLH4S9y+z3/SMpGfQYXxsX8e7/9rPhwMmFLrs2CWR8n1gu7xKJp5umGhARqQkKNyLnKu8YLJ1un2HYsILFk5yed/KO6SreW5ROes5GANwtJq7oEsX4PrF0iwlybs0iIg2Awo24NmuJfcK8jR8DJnDztC9vYPEove0OFs+/3C593HG79DE3z5O3i/Nh7X8g/zgAx5sNY6Z5HB/9bKLEdgiA8ABPxsY1Y0zvpjT293TiN0FEpGFRuBHXVZgNn90Eu36ssZfI8G/Nc9zEJ9ubl+4x6BUbzPg+sQztGIG7RVc9iYjUNoUbcU1ZR+Dj6yB5M7h5w8hZEN4ZrIVQUmSfIfi0t4ugpPC0t/MLCtiTksHnx5rxblo/rFjwdDNzZbcoxsXH0ilaswiLiDiTwo24npQ/4aPrIOsg+DaGv8+H6B7n/bRbj2Qx9+c9LNxwcgHL6CBvbriwGaN7xWhVbhGROkLhRlzLnqUw/0YozIJGreGGBRAce85PZxgGK3cdZc7Pe1i+I82xv3dsCDdf1JxLOoRj0QKWIiJ1isKNuI4Nn8DCe8BWAs36wqgPwSfknJ6q2Grj602HmbN8L1uPZAFgNsHwTpHc2q853ZsGV2flIiJSjRRupP4zDFj2PCx9xn6/0zUw8g371U1VlFVQzLy1Sby9Yh/JWfZZhL3dLYzqFcPNfZvTtJEWsBQRqesUbqR+sxbDV/fCho/s9y/6BwyaCuaqXaV0OCOfd1bu5ZO1B8gpLAEg1M+TCX1jGRvXlCAfjacREakvFG6k/irIgk/HwZ4lYDLD5TOg581Veoo/DmXy5s97+HrTEccg4VZhftzWrwVXdo/SLMIiIvWQwo3UT5mH7FdEpW4Bd1+47l1oc2mlDjUMg2U70pj78x5W7jrq2B/fohG39W/BgDaNMWuQsIhIvaVwI/VP8mZ7sMk+An7h8PdPIarbWQ8rLLGycMNh3vx5L9tTsgH7qtyXd45kYr8WdG6i+WlERFyBwo3UL7sS4dPxUJQNjdvB2M8gqOkZD8kpLOH91ft4d+U+UrMLAfD1sDC6d1Mm9I2lSbAGCYuIuBKFG6k/1n9gHzxsWCG2n/1Sb++gMx6SnlPI2Lm/OHpqwgM8mdC3OWN6NyXQ270WihYRkdqmcCN1n2HAkqdh+Qv2+11Gwd9ety9qeQZp2YX8fe4adqbmEObvyQPD2jGiaxQeblrvSUTElSncSN1WUgQLJ8Gmefb7/f8FFz8MpjMP+E3NLuDvc39hV2oOEQFefHLbhTQP9a2FgkVExNkUbqTuys+A+TfAvp/BZIERM+GCcWc9LDWrgDFz17A7LZfIQC8+mXghsQo2IiINhsKN1E0ZSfYrotK2gYcfXP8etBpy1sNSsgoYM2cNe9JziQq099g0a6RgIyLSkDh98MGsWbOIjY3Fy8uLuLg41q5de8b2GRkZ3H333URGRuLp6UmbNm345ptvaqlaqRWHN8CbQ+zBxj8SJnxbqWCTnFnA6NJgEx3kzbzb4hVsREQaIKf23MyfP5+EhARmz55NXFwcM2fOZOjQoWzfvp2wsLBy7YuKirjkkksICwtjwYIFREdHs3//foKCgmq/eKkZSWvgg6uhOBfCOsLYTyGwyVkPO5KZz5g5a9h3NK802FxITIgu8RYRaYhMhmEYznrxuLg4evXqxeuvvw6AzWYjJiaGSZMm8eCDD5ZrP3v2bF544QW2bduGu/u5XcablZVFYGAgmZmZBAQEnFf9Us0O/w7v/Q0Ks6B5f/ul3l5nn1jvcEY+Y+auYf/RPJoE24ON5q4REXEtVfn8dtppqaKiItatW8eQISdPN5jNZoYMGcLq1asrPGbhwoXEx8dz9913Ex4eTqdOnXjmmWewWq2nfZ3CwkKysrLKbFIHpfwJH1xlDzbN+sKY+ZUKNgeP5zFqzmr2H80jJsSb+bfHK9iIiDRwTgs36enpWK1WwsPDy+wPDw8nOTm5wmP27NnDggULsFqtfPPNNzz66KPMmDGDp5566rSvM336dAIDAx1bTExMtb4PqQZHd8P7V0L+cYjuCX+fDx5nDygHjuUxes4aDhzLp1kjH+bfFk90kHctFCwiInWZ0wcUV4XNZiMsLIw5c+bQo0cPRo0axcMPP8zs2bNPe8yUKVPIzMx0bAcOHKjFiuWsMpLsp6JyUyG8M9ywADz9z3rYiWBz8Hg+sY18mHfbhUQp2IiICE4cUBwaGorFYiElJaXM/pSUFCIiIio8JjIyEnd3dywWi2Nf+/btSU5OpqioCA+P8jPWenp64unpWb3FS/XITrYHm6yD0Kg13PgFeAef9bCko3mMmbuGQxn5NA/15ZOJFxIR6FULBYuISH3gtJ4bDw8PevToQWJiomOfzWYjMTGR+Pj4Co/p27cvu3btwmazOfbt2LGDyMjICoON1GG5R+2noo7vhaBmMH4h+DU+62H7j+Yyes5qDmXk0yLUl3m3KdiIiEhZTj0tlZCQwNy5c3nvvffYunUrd955J7m5uUyYMAGAcePGMWXKFEf7O++8k2PHjnHvvfeyY8cOFi1axDPPPMPdd9/trLcg5yI/Az4YWTqPTZQ92AREnfWwfem5jJ6zhsOZBbRsbA824QEKNiIiUpZT57kZNWoUaWlpTJ06leTkZLp168bixYsdg4yTkpIwm0/mr5iYGL777jv+8Y9/0KVLF6Kjo7n33nt54IEHnPUWpKoKc+wzDydvAt/G9mATHHvWw/am5zJmzhqSswpoFebHxxPjCPNXsBERkfKcOs+NM2ieGycqzrcHm30/g1cQ3LQIIjqd9bDdaTn8fe4aUrIKaR3mx8cTL6Sxv8ZRiYg0JFX5/NbaUlI7Sorg03H2YOPhDzd+XqlgsyvVHmxSswtpG+7PRxPjCPVTsBERkdNTuJGaZy2Bz2+Fnd+Dm7d9HpvoHmc9bFdqNmPm/kJadiHtIvz56NY4GinYiIjIWSjcSM2y2WDhPfDnl2DxgNEfQmzfsx62M8UebNJz7MHm44kXEuKrK+JEROTsFG6k5hgGfDMZNn4CJgtc926lVvfenpzN3+eu4WhuER0iA/jo1jiCFWxERKSSFG6kZhgG/PAo/PYWYIKr50C7y8962G/7jnH7B+s4mltExyh7sAnyUbAREZHKU7iRmrHsOVj1mv32316Fztee9ZCPf0li2sI/KLYadI4O5INbeivYiIhIlSncSPVb+SosnW6/Pew5uGDcGZsXldh47KstfPxLEgCXd47kheu64OOhX08REak6fXpI9fr1TfvpKIDBU+HCO87YPDW7gLs+XM9v+49jMsHkS9ty18CWmEymWihWRERckcKNVJ8Nn8Cif9pv9/unfTuDjQcyuP2DdSRnFeDv5caro7tzcbuwWihURERcmcKNVI8t/4Mv77LfjrsDBj16xuYL1h3koS82U1Rio1WYH3Nu7EGLxn41X6eIiLg8hRs5fzu+g//eAoYNut8IQ6fDaU4rFVttPL1oK++u2gfAkPbhvDyqK/5e7rVYsIiIuDKFGzk/e5fD/BvBVgKdroURr4C54sXmj+UWcfdH61m95ygA9w5uzb2DW2M2a3yNiIhUH4UbOXepW2HeWLAWQtvL4arZYLZU2HTL4Uxue38dhzLy8fWwMOP6bgzrFFHLBYuISEOgcCPnJjsFProeCrOgaR+49m2wVHxqaeHGw9y/YCMFxTZiG/kwZ1xP2oT713LBIiLSUCjcSNUV5cEnoyEzCUJawuiPwN2rXDOrzeD577bxn2V7ABjQpjGvju5OoI/G14iISM1RuJGqsdngi9vg8HrwDoGxn4FPSLlmmXnFTJr3O8t3pAFwx4CW/GtoWywaXyMiIjVM4Uaq5sdpsPWr0hW+P4ZGLcs12ZGSzcT3f2P/0Ty83M08f21X/tY1ygnFiohIQ6RwI5X329uw6lX77ZFvQLP4ck0W/5FMwqcbyCuyEh3kzZxxPegYFVjLhYqISEOmcCOVs+tHWDTZfvviR8othGmzGcxM3MmriTsBiG/RiFljLyDEVwtfiohI7VK4kbNL2QKf3gSGFbr+HfpPLvNwdkEx/5i/gR+3pgIwoW8sD13WHndLxfPdiIiI1CSFGzmz7GT7Jd9F2RDbzz5J3ymzD+9Nz+XW935ld1ouHm5mnrmqM9f2aOLEgkVEpKFTuJHTK8qFj0dB1kFo1Bqufx/cTp5myi4o5qZ31rL/aB4RAV7858YedI0Jcl69IiIiKNzI6dis8N+JcGQD+DSCsZ+WueTbMAwe/uIP9h/NIzrImy/u7kOYf/m5bkRERGqbBkVIxb5/FLYvAosnjP4EQlqUefjT3w6wcONhLGYTr47ppmAjIiJ1hsKNlLd2LqyZZb991RvQNK7MwztTspm2cAsA/7y0DT2alZ/ET0RExFkUbqSsHd/Dt/fbbw+eCp2uKfNwQbGVez7+nYJiG/1ah3JH//KT+ImIiDiTwo2cdGQTLJgAhg263wAXJZRr8vhXf7I9JZtQP09eur4bZi2nICIidYzCjdhlHbZfGVWUA80HwBUzy1zyDfD1psN8sjYJkwlmjupGY39P59QqIiJyBgo3AoU58PH1kH0YQtvaL/m2lF25O+loHlP+uxmAuwa25KLWoc6oVERE5KwUbho6mxUW3AzJm8G3sf2Sb++gMk2KSmxM+mQ92YUl9GwWzD+GtHFOrSIiIpWgcNPQLZ4CO78DNy8YMw+CY8s1eeG7bWw8mEmgtzuvjOmOm5ZVEBGROkyfUg3Zmtmw9j/221f9B5r0LNfkp20pzP15LwDPX9uF6CDv2qxQRESkyhRuGqrt38J3U+y3hzwOHUeWa5KcWcDkzzYBcFOfWIZ2jKjFAkVERM6Nwk1DdHiDfZyNYYMLxkPfe8s1sdoM7pv/O8dyi+gYFcCUy9rVfp0iIiLnQOGmock8aL/kuzgPWlwMl88od8k3wGs/7WTNnmP4eFh4bUx3PN0sTihWRESk6hRuGpKiPHuwyUmGxu3h+vfKXfINsGbPUV5N3AnA01d1okVjv9quVERE5Jwp3DQkS5+BlD/AN8x+ybdXYLkmx3KLuHfe79gMuLZHE67q3sQJhYqIiJy7OhFuZs2aRWxsLF5eXsTFxbF27drTtn333XcxmUxlNi8vrUh9Vod/h9Wli2Fe+ToENS3XxDAMJn+2kZSsQlo09uWJKzvWcpEiIiLnz+nhZv78+SQkJDBt2jTWr19P165dGTp0KKmpqac9JiAggCNHjji2/fv312LF9ZC1GBZOsg8g7nQttBlaYbO3Vuzlp22peLiZmfX3C/DxcKvlQkVERM6f08PNSy+9xMSJE5kwYQIdOnRg9uzZ+Pj48Pbbb5/2GJPJREREhGMLDw8/bdvCwkKysrLKbA3O6tftMxB7B8OwZytssvFABs8t3gbAo1d0oH1kQG1WKCIiUm2cGm6KiopYt24dQ4YMcewzm80MGTKE1atXn/a4nJwcmjVrRkxMDFdeeSVbtmw5bdvp06cTGBjo2GJiYqr1PdR5R3fD0tJAM/QZ8GtcrklWQTGTPvmdYqvB8E4R3BBX/pSViIhIfeHUcJOeno7Vai3X8xIeHk5ycnKFx7Rt25a3336bL7/8kg8//BCbzUafPn04ePBghe2nTJlCZmamYztw4EC1v486yzDgq3uhpABaDISuYypoYvDQ55tJOpZHdJA3z17TBVMFl4aLiIjUF/VuUEV8fDzx8fGO+3369KF9+/b85z//4cknnyzX3tPTE09Pz9osse74/UPY9zO4ecMVMyucz2berwf4etMR3MwmXvt7dwK9y18aLiIiUp84tecmNDQUi8VCSkpKmf0pKSlERFRuqn93d3e6d+/Orl27aqLE+is7Bb5/xH774ocgpHm5JjtSsnlsof2U3uShbbmgaXBtVigiIlIjnBpuPDw86NGjB4mJiY59NpuNxMTEMr0zZ2K1Wtm8eTORkZE1VWb9tPgBKMiAyK5w4V3lHs4vsnL3R+spLLHRv01jbuvXovZrFBERqQFOPy2VkJDA+PHj6dmzJ71792bmzJnk5uYyYcIEAMaNG0d0dDTTp08H4IknnuDCCy+kVatWZGRk8MILL7B//35uvfVWZ76NumX7t7DlCzBZ4G+vgaX8j/nxr7awMzWHxv6evHR9V8xmjbMRERHX4PRwM2rUKNLS0pg6dSrJycl069aNxYsXOwYZJyUlYTaf7GA6fvw4EydOJDk5meDgYHr06MGqVavo0KGDs95C3VKQBYv+ab/d5x57z81fLNx4mHm/HsBkgpmjuhHq10DHJImIiEsyGYZhOLuI2pSVlUVgYCCZmZkEBLjgXC6LJsOvcyG4Ody5Cjx8yjy8/2gul7+6gpzCEiYNasU/L23rpEJFREQqryqf306fxE+qUdIv8Oub9tsjXikXbIpKbEz65HdyCkvoFRvMvYNbO6FIERGRmqVw4ypKCu1LLGBA9xugxYByTT7+ZT+bDmYS6O3OK6O742bRj19ERFyPPt1cxYqXIX07+DaGS8rP95NVUMwriTsBuH9YW6KCvGu7QhERkVqhcOMKUrfB8hftt4c/Dz4h5ZrMXrqb43nFtGzsy6ieDWwJChERaVAUbuo7mw2++j+wFUOb4dDxqnJNjmTm89aKvQA8OLy9TkeJiIhL06dcfffbW3DgF/Dwg8tfrHCJhZe+30FhiY3esSEMaR/mhCJFRERqj8JNfZZ5EH583H57yGMQ2KRck23JWSxYb19UdMpl7bQopoiIuDyFm/rKMOxz2hRlQ5Pe0POWCptN/2YbhgGXd46ku9aOEhGRBkDhpr7683+w41swu9uXWDCX/1Gu2JnOsh1puFtM3D9Mk/WJiEjDoHBTH+Udg2/+Zb/d758Q1q5cE5vNYPq3WwEYG9eMZo18a7NCERERp1G4qY9+eBRy0yC0LfRLqLDJwo2H2XI4C39PNyYNalXLBYqIiDiPwk19s2cZ/P4hYLKfjnIrv+hlQbGVF77bDsAdA1vSSAtjiohIA6JwU58U58NX99pv97oFmsZV2Oz91fs4lJFPZKAXt1zUvBYLFBERcT6Fm/pk6bNwfC/4R8HgaRU2ycgr4vWfdgGQcEkbvNwttVmhiIiI0ync1BdHNsKq1+y3L58BXhUv9z5ryS6yCkpoF+HP1ReUn/dGRETE1Snc1AfWElj4f2BY7csrtLuswmYHjuXx3qr9ADw4vB0WsybsExGRhkfhpj745Q04sgG8AmHYc6dt9uL32ymy2ujbqhED2jSuvfpERETqEIWbuu7YXvjpafvtS58G//AKm20+mMmXGw4DMGV4ey2zICIiDZbbuRyUm5vLs88+S2JiIqmpqdhstjKP79mzp1qKa/AMA77+B5TkQ/P+0P2G0zQzeOYb+4R9V3WPplN0YG1WKSIiUqecU7i59dZbWbZsGTfeeCORkZHqJagpG+fBniXg5gVXzKxwxW+ApTvSWL3nKB4WM/+8tE3t1igiIlLHnFO4+fbbb1m0aBF9+/at7nrkhKI8+PEx++0BD0CjlhU2s9oMnv1mGwA39Y2lSbBPLRUoIiJSN53TmJvg4GBCQkKquxY51do5kJMMQU0h/p7TNvvvuoNsT8km0NuduwdqmQUREZFzCjdPPvkkU6dOJS8vr7rrEYD8DFjxsv32wIfAzaPiZkVWZvxgX2Zh0qBWBPq411KBIiIiddc5nZaaMWMGu3fvJjw8nNjYWNzdy36orl+/vlqKa7BWvQYFGdC4PXS5/rTN3l65l5SsQpoEe3NjfLPaq09ERKQOO6dwM3LkyGouQxxyUmHNG/bbgx4Bc8XLJxzNKeSNpbsB+NfQtni6aZkFEREROMdwM21axesaSTVY/iIU50J0D2h3+WmbvfbTLnIKS+gUHcCILlG1WKCIiEjdds6T+GVkZPDmm28yZcoUjh07BthPRx06dKjaimtwju+H39623x489bSXfu9Nz+XDNfZlFh4a3h6zllkQERFxOKeem02bNjFkyBACAwPZt28fEydOJCQkhM8//5ykpCTef//96q6zYVj6LNiKofkAaDHwtM1e+G4bJTaDi9s2pk+r0NqrT0REpB44p56bhIQEbrrpJnbu3ImXl5dj/2WXXcby5currbgGJXUbbJpnvz349Kf91icd55vNyZhN8ODw9rVUnIiISP1xTuHm119/5fbbby+3Pzo6muTk5PMuqkFa8hQYNmh3BTTpUWETwzCYXrrMwrU9mtA2wr82KxQREakXzinceHp6kpWVVW7/jh07aNxYq1FX2aF1sPUrMJlh0KOnbfb9nyn8uu84Xu5mEi5pW4sFioiI1B/nFG7+9re/8cQTT1BcXAyAyWQiKSmJBx54gGuuuaZaC2wQEp+wf+0yGsLaVdik2GrjuW/tyyzcelELIgK9KmwnIiLS0J1TuJkxYwY5OTmEhYWRn5/PgAEDaNWqFf7+/jz99NPVXaNr27MM9iwFszsMfPC0zeb/eoA96bmE+Hpw+4AWtVefiIhIPXNOV0sFBgbyww8/sHLlSjZu3EhOTg4XXHABQ4YMwTCM6q7RdRkGJD5uv93zZgiueJbhnMISZv64A4B7B7fG30vLLIiIiJzOOYWbF154gX/961/07du3zMrgVquVG264gU8++aTaCnRp2xbZx9u4+0L/yadtNnf5HtJzioht5MOY3k1rsUAREZH655xOS73wwgu89dZbZfZZrVZGjx7Nhg0bqqMu12ezwk9P2m9feCf4hVXYLDWrgDnL9wBw/7B2eLid87yLIiIiDcI5fVIuWrSIyZMns2DBAgBKSkq47rrr2LJlC0uWLKny882aNYvY2Fi8vLyIi4tj7dq1lTpu3rx5mEym+rnW1aZPIW0beAVBn0mnbfbyjzvJL7bSvWkQwztF1F59IiIi9dQ5hZtevXrx3//+l5tvvpmFCxdyzTXXsH37dpYsWUJERNU+gOfPn09CQgLTpk1j/fr1dO3alaFDh5KamnrG4/bt28fkyZPp16/fubwF5yopgqXP2G9f9A/wDqqw2a7UbOb/mgTAQ5e1x3Sa5RhERETkpHM+xzFo0CDef/99rrnmGvbu3cuyZcsIDw+v8vO89NJLTJw4kQkTJtChQwdmz56Nj48Pb7/99mmPsVqtjB07lscff5wWLerhlUPr3oWMJPCLgN63nbbZK4m7sBlwaYdwesWG1F59IiIi9VilBxRfffXVFe5v3LgxQUFB3HbbyQ/pzz//vFLPWVRUxLp165gyZYpjn9lsZsiQIaxevfq0xz3xxBOEhYVxyy238PPPP5/xNQoLCyksLHTcr2jywVpVlAvLX7DfHvAv8PCpsFlhiZWftqYAcNfFrWqrOhERkXqv0uEmMDCwwv1Dhw495xdPT0/HarWW6/EJDw9n27ZtFR6zYsUK3nrrrUoPXJ4+fTqPP/74OddY7da8AbmpEBwL3cedttnavcfILbLS2N+TLtEVf+9FRESkvEqHm3feeacm66iU7OxsbrzxRubOnUtoaOVWw54yZQoJCQmO+1lZWcTExNRUiWeWdwxWvmq/ffHD4OZx2qaJW+1jjga1DcNs1lgbERGRyjqneW5OSEtLY/v27QC0bdu2yutKhYaGYrFYSElJKbM/JSWlwoHJu3fvZt++fYwYMcKxz2azAeDm5sb27dtp2bJlmWM8PT3x9PSsUl01ZtWrUJgJYR2h07WnbWYYBonb7N+Twe0rvkRcREREKnZOA4pzc3O5+eabiYyMpH///vTv35+oqChuueUW8vLyKv08Hh4e9OjRg8TERMc+m81GYmIi8fHx5dq3a9eOzZs3s2HDBsf2t7/9jYsvvpgNGzY4r0emMrKTYc1s++3Bj4L59N/63Wk5HDiWj4ebmb6tKtdDJSIiInbnFG4SEhJYtmwZX331FRkZGWRkZPDll1+ybNky/vnPf1b5uebOnct7773H1q1bufPOO8nNzWXChAkAjBs3zjHg2MvLi06dOpXZgoKC8Pf3p1OnTnh4nP40j9MtfwFK8qFJb2gz7IxNfyw9JRXfohG+nufVuSYiItLgnNMn53//+18WLFjAwIEDHfsuu+wyvL29uf7663njjTcq/VyjRo0iLS2NqVOnkpycTLdu3Vi8eLFjkHFSUhLmM/Ry1AvH9tov/wYYMg3OMl/NT6XhRqekREREqu6cwk1eXl6Fc9qEhYVV6bTUCffccw/33HNPhY8tXbr0jMe+++67VX69Wrd0OthKoOVgiL3ojE0z8or4bf8xAC5uq3AjIiJSVefUJRIfH8+0adMoKChw7MvPz+fxxx+vcKxMg5ayxb7UAsDgqWdtvmxHGjYD2ob7ExNS8Rw4IiIicnrn1HMzc+ZMhg0bRpMmTejatSsAGzduxMvLi++++65aC6z3fnoKMKDDSIjqdtbmiTolJSIicl7OKdx07tyZnTt38tFHHzkm2xszZgxjx47F29u7Wgus1w6she3fgMkCgx45a/Niq42l2xVuREREzsc5hZvly5fTp08fJk6cWGZ/SUkJy5cvp3///tVSXL1mGJD4hP12t79DaOuzHrJu/3GyCkoI9nGnW0xwDRcoIiLims5pzM3FF1/MsWPHyu3PzMzk4osvPu+iXMLun2Dfz2DxhIEPVuqQn7bZe20ubhuGRbMSi4iInJNzCjeGYWCq4HLmo0eP4uvre95F1Xun9tr0uhUCm1TqsMTShTIH6ZSUiIjIOavSaakTK4ObTCZuuummMssaWK1WNm3aRJ8+faq3wvrozy/hyAbw8IN+CWdtDrAvPZfdabm4mU30b1O1ZSxERETkpCqFmxMrgxuGgb+/f5nBwx4eHlx44YXlxuE0ONaS0iukgPi7wbdyyyecOCXVu3kIAV7uNVWdiIiIy6tSuJk1axY+Pj7ExsYyefJknYKqyKZ5cHQneIdAfMUTE1bkxEKZg9rplJSIiMj5qNKYm9DQUK644goiIyPJzs6uqZrqr5JCWPqs/Xa/BPAKqNRh2QXF/LLHPkB7cPvyMz+LiIhI5VUp3GzdupWhQ4fy6aefEhsbS1xcHE8//TSbN2+uqfrql9/ehswD4B9lH0hcST/vTKfEZtAi1JfmoeoNExEROR9VCjfNmjVj0qRJ/Pjjj6SkpHDfffexefNm+vXrR4sWLbjvvvv46aefsFqtNVVv3VWYDctftN8e+AC4V34yQ81KLCIiUn3OebntwMBAxowZw7x580hLS+M///kPVquVCRMm0LhxYz766KPqrLPu2/o15KVDSEvodkOlD7PaDMesxIPa6ZSUiIjI+TqnGYr/yt3dnUsuuYRLLrmE1157jd9//52SkpLqeOr6o9sYCIgCwwaWyn9bNxzI4GhuEf5ebvSM1azEIiIi56tKPTfPP/88+fn5jvsrV66ksLDQcT87O5u77rqL7t2706tXr+qrsr5oMQBaVm2G5p9Kr5Ia0KYx7pZz7kgTERGRUlX6NJ0yZUqZq6SGDx/OoUOHHPfz8vL4z3/+U33VNQAabyMiIlK9qhRuDMM4432pmkMZ+WxLzsZsgoFtFG5ERESqg86DONGJWYl7NAsm2NfDydWIiIi4BoUbJ/rpxEKZukpKRESk2lT5aqk333wTPz8/AEpKSnj33XcJDbWvn6RZiysvr6iElbuPAhpvIyIiUp2qFG6aNm3K3LlzHfcjIiL44IMPyrWRs1u56yhFJTaaBHvTOszP2eWIiIi4jCqFm3379tVQGQ3PiUvAB7cLw2QyObkaERER11GlMTc//fQTHTp0ICsrq9xjmZmZdOzYkZ9//rnainNVhmGccgm4xtuIiIhUpyqFm5kzZzJx4kQCAsqvdh0YGMjtt9/OSy+9VG3Fuaoth7NIzS7Ex8NCXIsQZ5cjIiLiUqoUbjZu3MiwYcNO+/ill17KunXrzrsoV/dj6VVS/VqH4ulmcXI1IiIirqVK4SYlJQV3d/fTPu7m5kZaWtp5F+XqTsxvM1iXgIuIiFS7KoWb6Oho/vjjj9M+vmnTJiIjI8+7KFeWmlXApoOZAAxs19jJ1YiIiLieKoWbyy67jEcffZSCgoJyj+Xn5zNt2jSuuOKKaivOFS3Zbu+16RoTRJi/l5OrERERcT1VuhT8kUce4fPPP6dNmzbcc889tG3bFoBt27Yxa9YsrFYrDz/8cI0U6iocV0m108R9IiIiNaFK4SY8PJxVq1Zx5513MmXKFMfCmSaTiaFDhzJr1izCwzWO5HQKiq2s2JUOwCCFGxERkRpR5eUXmjVrxjfffMPx48fZtWsXhmHQunVrgoODa6I+l7Jmz1HyiqyEB3jSMar85fQiIiJy/qocbk4IDg6mV69e1VmLyztxldSgduGalVhERKSGaFXwWlJmVmKdkhIREakxCje1ZEdKDocy8vF0M9O3VaizyxEREXFZCje1JLF0ocy+rULx9tCsxCIiIjVF4aaWnDglpaukREREapbCTS04llvE+qTjgMKNiIhITasT4WbWrFnExsbi5eVFXFwca9euPW3bzz//nJ49exIUFISvry/dunXjgw8+qMVqq27p9lQMA9pHBhAV5O3sckRERFya08PN/PnzSUhIYNq0aaxfv56uXbsydOhQUlNTK2wfEhLCww8/zOrVq9m0aRMTJkxgwoQJfPfdd7VceeUlll4CPqS9em1ERERqmsk4Mc2wk8TFxdGrVy9ef/11AGw2GzExMUyaNIkHH3ywUs9xwQUXcPnll/Pkk0+We6ywsJDCwkLH/aysLGJiYsjMzCQgoOYn0iu22rjgiR/ILizhi7v60L2pJjsUERGpqqysLAIDAyv1+e3UnpuioiLWrVvHkCFDHPvMZjNDhgxh9erVZz3eMAwSExPZvn07/fv3r7DN9OnTCQwMdGwxMTHVVn9l/LrvGNmFJTTy9aBrk6BafW0REZGGyKnhJj09HavVWm49qvDwcJKTk097XGZmJn5+fnh4eHD55Zfz2muvcckll1TYdsqUKWRmZjq2AwcOVOt7OJsTV0ld3C4Ms1mzEouIiNS0c15+wZn8/f3ZsGEDOTk5JCYmkpCQQIsWLRg4cGC5tp6ennh6etZ+kaVOLLmgWYlFRERqh1PDTWhoKBaLhZSUlDL7U1JSiIiIOO1xZrOZVq1aAdCtWze2bt3K9OnTKww3zrQnLYe96bm4W0z0a9PY2eWIiIg0CE49LeXh4UGPHj1ITEx07LPZbCQmJhIfH1/p57HZbGUGDdcVJ3ptLmzRCD/PetlJJiIiUu84/RM3ISGB8ePH07NnT3r37s3MmTPJzc1lwoQJAIwbN47o6GimT58O2AcI9+zZk5YtW1JYWMg333zDBx98wBtvvOHMt1EhzUosIiJS+5webkaNGkVaWhpTp04lOTmZbt26sXjxYscg46SkJMzmkx1Mubm53HXXXRw8eBBvb2/atWvHhx9+yKhRo5z1FiqUmV/Mr/uOAQo3IiIitcnp89zUtqpcJ38+vtp4mEmf/E6rMD9+TBhQY68jIiLSENSbeW5cma6SEhERcQ6FmxpgtRks2V4abtqHn6W1iIiIVCeFmxrwe9JxMvKKCfR254KmQc4uR0REpEFRuKkBJxbKHNi2MW4WfYtFRERqkz55a0DiVvukhLpKSkREpPYp3FSzA8fy2JGSg8VsYmAbhRsREZHapnBTzU5cJdWzWTCBPu5OrkZERKThUbipZifG2wxur14bERERZ1C4qUa5hSWs2X0UgEHtdAm4iIiIMyjcVKOfd6ZTZLXRrJEPLRv7OrscERGRBknhphr9tO3kVVImk8nJ1YiIiDRMCjfVxGYz+GlbGgBDNCuxiIiI0yjcVJPNhzJJzynEz9ONXrEhzi5HRESkwXJzdgGuwmwycUmHcAK83PFwU2YUERFxFoWbatK5SSBzx/V0dhkiIiINnroYRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUhRuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUhRuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcSp0IN7NmzSI2NhYvLy/i4uJYu3btadvOnTuXfv36ERwcTHBwMEOGDDljexEREWlYnB5u5s+fT0JCAtOmTWP9+vV07dqVoUOHkpqaWmH7pUuXMmbMGJYsWcLq1auJiYnh0ksv5dChQ7VcuYiIiNRFJsMwDGcWEBcXR69evXj99dcBsNlsxMTEMGnSJB588MGzHm+1WgkODub1119n3LhxZ22flZVFYGAgmZmZBAQEnHf9IiIiUvOq8vnt1J6boqIi1q1bx5AhQxz7zGYzQ4YMYfXq1ZV6jry8PIqLiwkJCanw8cLCQrKysspsIiIi4rqcGm7S09OxWq2Eh4eX2R8eHk5ycnKlnuOBBx4gKiqqTEA61fTp0wkMDHRsMTEx5123iIiI1F1OH3NzPp599lnmzZvHF198gZeXV4VtpkyZQmZmpmM7cOBALVcpIiIitcnNmS8eGhqKxWIhJSWlzP6UlBQiIiLOeOyLL77Is88+y48//kiXLl1O287T0xNPT89qqVdERETqPqf23Hh4eNCjRw8SExMd+2w2G4mJicTHx5/2uOeff54nn3ySxYsX07Nnz9ooVUREROoJp/bcACQkJDB+/Hh69uxJ7969mTlzJrm5uUyYMAGAcePGER0dzfTp0wF47rnnmDp1Kh9//DGxsbGOsTl+fn74+fk57X2IiIhI3eD0cDNq1CjS0tKYOnUqycnJdOvWjcWLFzsGGSclJWE2n+xgeuONNygqKuLaa68t8zzTpk3jscceq83SRUREpA5y+jw3tU3z3IiIiNQ/9WaeGxEREZHqpnAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSlODzezZs0iNjYWLy8v4uLiWLt27WnbbtmyhWuuuYbY2FhMJhMzZ86svUJFRESkXnBquJk/fz4JCQlMmzaN9evX07VrV4YOHUpqamqF7fPy8mjRogXPPvssERERtVytiIiI1AcmwzAMZ714XFwcvXr14vXXXwfAZrMRExPDpEmTePDBB894bGxsLPfddx/33XffGdsVFhZSWFjouJ+VlUVMTAyZmZkEBASc93sQERGRmpeVlUVgYGClPr+d1nNTVFTEunXrGDJkyMlizGaGDBnC6tWrq+11pk+fTmBgoGOLiYmptucWERGRusdp4SY9PR2r1Up4eHiZ/eHh4SQnJ1fb60yZMoXMzEzHduDAgWp7bhEREal73JxdQE3z9PTE09PT2WWIiIhILXFaz01oaCgWi4WUlJQy+1NSUjRYWERERM6Z08KNh4cHPXr0IDEx0bHPZrORmJhIfHy8s8oSERGRes6pp6USEhIYP348PXv2pHfv3sycOZPc3FwmTJgAwLhx44iOjmb69OmAfRDyn3/+6bh96NAhNmzYgJ+fH61atXLa+xAREZG6w6nhZtSoUaSlpTF16lSSk5Pp1q0bixcvdgwyTkpKwmw+2bl0+PBhunfv7rj/4osv8uKLLzJgwACWLl1a2+WLiIhIHeTUeW6coSrXyYuIiEjdUC/muRERERGpCQo3IiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lIUbkRERMSlKNyIiIiIS6kT4WbWrFnExsbi5eVFXFwca9euPWP7zz77jHbt2uHl5UXnzp355ptvaqlSERERqeucHm7mz59PQkIC06ZNY/369XTt2pWhQ4eSmppaYftVq1YxZswYbrnlFn7//XdGjhzJyJEj+eOPP2q5chEREamLTIZhGM4sIC4ujl69evH6668DYLPZiImJYdKkSTz44IPl2o8aNYrc3Fy+/vprx74LL7yQbt26MXv27LO+XlZWFoGBgWRmZhIQEFB9b0RERERqTFU+v91qqaYKFRUVsW7dOqZMmeLYZzabGTJkCKtXr67wmNWrV5OQkFBm39ChQ/nf//5XYfvCwkIKCwsd9zMzMwH7N0lERETqhxOf25Xpk3FquElPT8dqtRIeHl5mf3h4ONu2bavwmOTk5ArbJycnV9h++vTpPP744+X2x8TEnGPVIiIi4izZ2dkEBgaesY1Tw01tmDJlSpmeHpvNxrFjx2jUqBEmk6laXysrK4uYmBgOHDigU151iH4udZd+NnWTfi51V0P+2RiGQXZ2NlFRUWdt69RwExoaisViISUlpcz+lJQUIiIiKjwmIiKiSu09PT3x9PQssy8oKOjci66EgICABvdLVx/o51J36WdTN+nnUnc11J/N2XpsTnDq1VIeHh706NGDxMRExz6bzUZiYiLx8fEVHhMfH1+mPcAPP/xw2vYiIiLSsDj9tFRCQgLjx4+nZ8+e9O7dm5kzZ5Kbm8uECRMAGDduHNHR0UyfPh2Ae++9lwEDBjBjxgwuv/xy5s2bx2+//cacOXOc+TZERESkjnB6uBk1ahRpaWlMnTqV5ORkunXrxuLFix2DhpOSkjCbT3Yw9enTh48//phHHnmEhx56iNatW/O///2PTp06OestOHh6ejJt2rRyp8HEufRzqbv0s6mb9HOpu/SzqRynz3MjIiIiUp2cPkOxiIiISHVSuBERERGXonAjIiIiLkXhRkRERFyKwk01mTVrFrGxsXh5eREXF8fatWudXVKD99hjj2Eymcps7dq1c3ZZDdLy5csZMWIEUVFRmEymcmvBGYbB1KlTiYyMxNvbmyFDhrBz507nFNuAnO3nctNNN5X7Gxo2bJhzim1Apk+fTq9evfD39ycsLIyRI0eyffv2Mm0KCgq4++67adSoEX5+flxzzTXlJrhtyBRuqsH8+fNJSEhg2rRprF+/nq5duzJ06FBSU1OdXVqD17FjR44cOeLYVqxY4eySGqTc3Fy6du3KrFmzKnz8+eef59VXX2X27Nn88ssv+Pr6MnToUAoKCmq50oblbD8XgGHDhpX5G/rkk09qscKGadmyZdx9992sWbOGH374geLiYi699FJyc3Mdbf7xj3/w1Vdf8dlnn7Fs2TIOHz7M1Vdf7cSq6xhDzlvv3r2Nu+++23HfarUaUVFRxvTp051YlUybNs3o2rWrs8uQvwCML774wnHfZrMZERERxgsvvODYl5GRYXh6ehqffPKJEypsmP76czEMwxg/frxx5ZVXOqUeOSk1NdUAjGXLlhmGYf/7cHd3Nz777DNHm61btxqAsXr1ameVWaeo5+Y8FRUVsW7dOoYMGeLYZzabGTJkCKtXr3ZiZQKwc+dOoqKiaNGiBWPHjiUpKcnZJclf7N27l+Tk5DJ/Q4GBgcTFxelvqA5YunQpYWFhtG3bljvvvJOjR486u6QGJzMzE4CQkBAA1q1bR3FxcZm/mXbt2tG0aVP9zZRSuDlP6enpWK1Wx4zKJ4SHh5OcnOykqgQgLi6Od999l8WLF/PGG2+wd+9e+vXrR3Z2trNLk1Oc+DvR31DdM2zYMN5//30SExN57rnnWLZsGcOHD8dqtTq7tAbDZrNx33330bdvX8dM/MnJyXh4eJRbBFp/Myc5ffkFkZoyfPhwx+0uXboQFxdHs2bN+PTTT7nlllucWJlI/TB69GjH7c6dO9OlSxdatmzJ0qVLGTx4sBMrazjuvvtu/vjjD40XrCL13Jyn0NBQLBZLuVHqKSkpREREOKkqqUhQUBBt2rRh165dzi5FTnHi70R/Q3VfixYtCA0N1d9QLbnnnnv4+uuvWbJkCU2aNHHsj4iIoKioiIyMjDLt9TdzksLNefLw8KBHjx4kJiY69tlsNhITE4mPj3diZfJXOTk57N69m8jISGeXIqdo3rw5ERERZf6GsrKy+OWXX/Q3VMccPHiQo0eP6m+ohhmGwT333MMXX3zBTz/9RPPmzcs83qNHD9zd3cv8zWzfvp2kpCT9zZTSaalqkJCQwPjx4+nZsye9e/dm5syZ5ObmMmHCBGeX1qBNnjyZESNG0KxZMw4fPsy0adOwWCyMGTPG2aU1ODk5OWX+t7937142bNhASEgITZs25b777uOpp56idevWNG/enEcffZSoqChGjhzpvKIbgDP9XEJCQnj88ce55ppriIiIYPfu3dx///20atWKoUOHOrFq13f33Xfz8ccf8+WXX+Lv7+8YRxMYGIi3tzeBgYHccsstJCQkEBISQkBAAJMmTSI+Pp4LL7zQydXXEc6+XMtVvPbaa0bTpk0NDw8Po3fv3saaNWucXVKDN2rUKCMyMtLw8PAwoqOjjVGjRhm7du1ydlkN0pIlSwyg3DZ+/HjDMOyXgz/66KNGeHi44enpaQwePNjYvn27c4tuAM70c8nLyzMuvfRSo3Hjxoa7u7vRrFkzY+LEiUZycrKzy3Z5Ff1MAOOdd95xtMnPzzfuuusuIzg42PDx8TGuuuoq48iRI84ruo4xGYZh1H6kEhEREakZGnMjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIg2CyWTif//7n7PLEJFaoHAjIjXqpptuwmQylduGDRvm7NKq5NdffyUqKgqAw4cP4+3tTVFRkZOrEpGKaOFMEalxw4YN45133imzz9PT00nVnJvVq1fTt29fAH7++Wd69uyJh4eHk6sSkYqo50ZEapynpycRERFltuDgYMfjJpOJN954g+HDh+Pt7U2LFi1YsGBBmefYvHkzgwYNwtvbm0aNGnHbbbeRk5NTps3bb79Nx44d8fT0JDIyknvuuafM4+np6Vx11VX4+PjQunVrFi5cWOn3sGrVKke4WbFiheO2iNQ9CjciUic8+uijXHPNNWzcuJGxY8cyevRotm7dCkBubi5Dhw4lODiYX3/9lc8++4wff/yxTHh54403uPvuu7ntttvYvHkzCxcupFWrVmVe4/HHH+f6669n06ZNXHbZZYwdO5Zjx46dtqYVK1YQFBREUFAQCxYs4OGHHyYoKIjZs2fz6quvEhQUxLPPPlsz3xAROXfOXpZcRFzb+PHjDYvFYvj6+pbZnn76aUcbwLjjjjvKHBcXF2fceeedhmEYxpw5c4zg4GAjJyfH8fiiRYsMs9lsJCcnG4ZhGFFRUcbDDz982joA45FHHnHcz8nJMQDj22+/Pe0x+fn5xt69e41vv/3WCA4ONvbs2WP89ttvhoeHh7F161Zj7969xvHjx6v0/RCRmqcxNyJS4y6++GLeeOONMvtCQkLK3I+Pjy93f8OGDQBs3bqVrl274uvr63i8b9++2Gw2tm/fjslk4vDhwwwePPiMdXTp0sVx29fXl4CAAFJTU0/b3svLi9jYWD799FOGDx9O8+bNWbVqFf369aNdu3ZnfC0RcR6FGxGpcb6+vuVOEVUnb2/vSrVzd3cvc99kMmGz2U7b3s/PD4DCwkLMZjNffvklRUVFGIaBn58f/fr149tvvz33wkWkRmjMjYjUCWvWrCl3v3379gC0b9+ejRs3kpub63h85cqVmM1m2rZti7+/P7GxsSQmJlZrTRs2bOC3337DYrGQmJjIhg0baNSoEZ9++ikbNmzgzTffrNbXE5HqoZ4bEalxhYWFJCcnl9nn5uZGaGio4/5nn31Gz549ueiii/joo49Yu3Ytb731FgBjx45l2rRpjB8/nscee4y0tDQmTZrEjTfeSHh4OACPPfYYd9xxB2FhYQwfPpzs7GxWrlzJpEmTzrnuVq1asWbNGsLDw7noootISkoiOzubESNG4Oamfz5F6ir9dYpIjVu8eDGRkZFl9rVt25Zt27Y57j/++OPMmzePu+66i8jISD755BM6dOgAgI+PD9999x333nsvvXr1wsfHh2uuuYaXXnrJcfz48eMpKCjg5ZdfZvLkyYSGhnLttdeed+1Lly6lf//+ACxbtoz4+HgFG5E6zmQYhuHsIkSkYTOZTHzxxReMHDnS2aWIiAvQmBsRERFxKQo3IiIi4lJ04lhEnE5nx0WkOqnnRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLuX/AT/pu6XwcL4cAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(history.history['masked_acc'], label='accuracy')\n",
"plt.plot(history.history['val_masked_acc'], label='val_accuracy')\n",
"plt.ylim([0, max(plt.ylim())])\n",
"plt.xlabel('Epoch #')\n",
"plt.ylabel('CE/token')\n",
"plt.legend()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mU3Ce8M6I3rz"
},
"source": [
"### Translate\n",
"\n",
"Now that the model is trained, implement a function to execute the full `text => text` translation. This code is basically identical to the [inference example](#inference) in the [decoder section](#the_decoder), but this also captures the attention weights."
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:10.189649Z",
"iopub.status.busy": "2023-12-07T12:24:10.189411Z",
"iopub.status.idle": "2023-12-07T12:24:10.195285Z",
"shell.execute_reply": "2023-12-07T12:24:10.194684Z"
},
"id": "mmgYPCVgEwp_"
},
"outputs": [],
"source": [
"#@title\n",
"@Translator.add_method\n",
"def translate(self,\n",
" texts, *,\n",
" max_length=50,\n",
" temperature=0.0):\n",
" # Process the input texts\n",
" context = self.encoder.convert_input(texts)\n",
" batch_size = tf.shape(texts)[0]\n",
"\n",
" # Setup the loop inputs\n",
" tokens = []\n",
" attention_weights = []\n",
" next_token, done, state = self.decoder.get_initial_state(context)\n",
"\n",
" for _ in range(max_length):\n",
" # Generate the next token\n",
" next_token, done, state = self.decoder.get_next_token(\n",
" context, next_token, done, state, temperature)\n",
" \n",
" # Collect the generated tokens\n",
" tokens.append(next_token)\n",
" attention_weights.append(self.decoder.last_attention_weights)\n",
" \n",
" if tf.executing_eagerly() and tf.reduce_all(done):\n",
" break\n",
"\n",
" # Stack the lists of tokens and attention weights.\n",
" tokens = tf.concat(tokens, axis=-1) # t*[(batch 1)] -> (batch, t)\n",
" self.last_attention_weights = tf.concat(attention_weights, axis=1) # t*[(batch 1 s)] -> (batch, t s)\n",
"\n",
" result = self.decoder.tokens_to_text(tokens)\n",
" return result"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "U4XufRntbbva"
},
"source": [
"Here are the two helper methods, used above, to convert tokens to text, and to get the next token:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:10.198316Z",
"iopub.status.busy": "2023-12-07T12:24:10.198091Z",
"iopub.status.idle": "2023-12-07T12:24:10.384098Z",
"shell.execute_reply": "2023-12-07T12:24:10.383514Z"
},
"id": "E5hqvbR5FUCD"
},
"outputs": [
{
"data": {
"text/plain": [
"'is it still home ? '"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"result = model.translate(['¿Todavía está en casa?']) # Are you still home\n",
"result[0].numpy().decode()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wQ1iU63cVgfs"
},
"source": [
"Use that to generate the attention plot:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:10.387418Z",
"iopub.status.busy": "2023-12-07T12:24:10.387171Z",
"iopub.status.idle": "2023-12-07T12:24:10.393730Z",
"shell.execute_reply": "2023-12-07T12:24:10.392990Z"
},
"id": "s5hQWlbN3jGF"
},
"outputs": [],
"source": [
"#@title\n",
"@Translator.add_method\n",
"def plot_attention(self, text, **kwargs):\n",
" assert isinstance(text, str)\n",
" output = self.translate([text], **kwargs)\n",
" output = output[0].numpy().decode()\n",
"\n",
" attention = self.last_attention_weights[0]\n",
"\n",
" context = tf_lower_and_split_punct(text)\n",
" context = context.numpy().decode().split()\n",
"\n",
" output = tf_lower_and_split_punct(output)\n",
" output = output.numpy().decode().split()[1:]\n",
"\n",
" fig = plt.figure(figsize=(10, 10))\n",
" ax = fig.add_subplot(1, 1, 1)\n",
"\n",
" ax.matshow(attention, cmap='viridis', vmin=0.0)\n",
"\n",
" fontdict = {'fontsize': 14}\n",
"\n",
" ax.set_xticklabels([''] + context, fontdict=fontdict, rotation=90)\n",
" ax.set_yticklabels([''] + output, fontdict=fontdict)\n",
"\n",
" ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
" ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
"\n",
" ax.set_xlabel('Input text')\n",
" ax.set_ylabel('Output text')"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:10.397174Z",
"iopub.status.busy": "2023-12-07T12:24:10.396757Z",
"iopub.status.idle": "2023-12-07T12:24:10.808896Z",
"shell.execute_reply": "2023-12-07T12:24:10.808187Z"
},
"id": "rrGawQv2eiA4"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmpfs/tmp/ipykernel_16852/3355722706.py:23: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_xticklabels([''] + context, fontdict=fontdict, rotation=90)\n",
"/tmpfs/tmp/ipykernel_16852/3355722706.py:24: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_yticklabels([''] + output, fontdict=fontdict)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3AAAALGCAYAAAAEDnMLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOT0lEQVR4nO3deZhWdcE//vc9DJspgwqoKIKYhqaAPq5ZirhnKW65K25PpuZWamYKbiGp33Ip7SkVl9JyTcolJdFcsRLN1DQExQQFE0ZB9vn94c+pCZBthsOZeb2u677inPO5z/3mPiHz5nzOOZW6urq6AAAAsMKrKjoAAAAAi0eBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASqK66ACs2Pr377/M+xg4cGCOOOKIRkgDAAAtmwLHpxo5cuQyvb9SqaRfv36NkgUAAFo6UyhZpMGDB2fevHlL9aqrqys6PjSq8ePH5+tf/3rWX3/9tG/fPq1atZrvVV3t38YAgKbhpwyAxfT6669n6623zvvvv5/Pf/7zmTlzZrp375527drl9ddfz+zZs9OnT5907Nix6KgAQDPlDByf6v7778+hhx5a2PthRXL++edn6tSpGTFiRJ5//vkkyVFHHZWXX34548aNy1577ZVp06bljjvuKDgpANBcVerMcQNYLGuvvXa23HLL3HPPPUmSqqqqDBo0KIMGDUqSzJw5M5tuuml23HHH/PSnPy0wKQDQXDkDxyL17NkzV155ZdExoHCTJ09Or1696perq6szffr0+uW2bdtml112yW9/+9si4gEALYACxyKNGzcuU6ZMKToGFK5Tp06ZNm1ag+Vx48Y1GFNdXe3PCwDQZBQ4gMW0wQYbZMyYMfXLW221VR588MG8/vrrSZJJkybljjvuyPrrr19URACgmVPgABbTHnvskUceeaT+DNupp56aDz74IL17986WW26ZDTfcMBMnTsw3v/nNYoMCAM2WAsdiqVQqRUeAwn3jG9/IyJEj06pVqyRJv379ctttt6V79+558cUXs8Yaa+TKK6/McccdV3BSAKC5chdKFqmqqiodO3ZcomdbVSqVBlPNAACAZafAsUhVVUt3onbevHmNnAQAAFo2UyhZLIMHD868efOW6AUAADQuBQ5gIaqqqlJdXZ1XX321frlVq1aLfFVXVxecHABorvyUAbAQ22+/fSqVSlZaaaUGywAARVHgABZi5MiRn7oMALC8mUJJo5s3b16GDRtWdAwAAGh2FDgWaezYsTnllFMWa+wvfvGLbLTRRjnmmGOaOBUsf4cffngeeuihuHkvAFAUjxFgsbz33nu5+uqr8+c//znV1dX50pe+lOOPPz7t27dPktx77705++yz88orryRJ9t1339x+++1FRoZGV1VVlUqlkjXXXDMHH3xwDjvssPTt27foWADQIrRq1WqZ9zFo0KCcd955jZCmOAoci/TOO+9kq622yltvvVV/5qFSqWS77bbLiBEjcuyxx+aWW25JkgwYMCCDBg1K7969i4wMTeK5557LTTfdlNtuuy3vvPNOKpVKNt544xxxxBE55JBDsvbaaxcdEQCaraqqqnTv3j09evRY4vfW1dXlsccey+DBgxU4mr+TTz45V199db785S9n4MCBSZLrr78+Dz74YLbbbrs8/vjj2X777XPFFVekT58+xYaF5WDevHl58MEHc8stt+Q3v/lNpk+fnqqqqvTr1y+HH3549ttvv6y88spFxwSAZqWqqmqZCtiyvn9F4S6ULNKDDz6YjTfeOL/97W/r1+23337ZZJNN8sQTT+SII45w0xJalKqqquyxxx7ZY489Mm3atNxxxx255ZZb8sgjj+SRRx7JiSeemA8//LDomABAM+QMHIu00kor5bjjjssVV1zRYP0pp5ySq6++OmPGjFmqU9nQnHz44Ye58sorc/7552fOnDmZO3du0ZEAoFmZOXNmqqurl/pauGV9/4rCGTgWacaMGVl99dXnW7/aaqslifJGi/XfUyk/+uijVCqV7LzzzkVHA4Bmp23btoW+f0WhwAEsoT/96U+55ZZbctttt2XSpEmpq6vLpptumsMPPzyHHHJIunbtWnREAKCZUuBYLI8//nh+8IMfzLcuSS699NIFPhfrzDPPXC7ZYHm56KKL8otf/CKvvvpq6urqstZaa+X000/P4Ycf7s6rALCc3XHHHfnVr36VZ599NpMnT06SdO7cOVtuuWUOOeSQDBgwoNiATcQ1cCxSVdWSP++9Uqm4Bohmp6qqKp/5zGeyzz775PDDD89OO+20VH8+AIClN2XKlOy777559NFHF3gSIfn4Z9H+/fvnzjvvTIcOHZZzwqblDByLdMMNNxQdAVYIN910U/bdd9+stNJKRUcBgBbryCOPzMiRI7PRRhvl29/+dvr161d/+cLbb7+dRx55JJdffnlGjBiRo446KnfeeWfBiRuXM3AAAEApPPHEE/nSl76UnXfeOffee2/atWu3wHEzZszIV77ylTzyyCN5/PHHs+222y7npE3H3B8a3axZs3LbbbcVHQOa1Ny5c/POO+/kzTffXOALAGh8t956a6qrq3PdddcttLwlSbt27XL99denqqoqt95663JM2PScgaPRPP/887nuuuvyi1/8IlOmTHENHM3Sn//853z3u9/NY489llmzZi1wTKVSyZw5c5ZzMqAl+eCDD3L11Vfn4Ycfzttvv52ZM2fON6ZSqWTMmDEFpIOms/XWW6d9+/YZOXLkYo3v169fZsyYkaeffrppgy1HroFjmUydOjW//OUvc9111+W5555LXV1d2rdvn0MOOaToaNDoRo8enS996Uuprq7OrrvumuHDh6dPnz5Zc80185e//CWTJk1Kv3790r1796KjAs3YpEmT8oUvfCFjxoxJhw4dUltbm5qamsyaNSsfffRRkqRr165p3bp1wUmh8Y0dOzYHHnjgYo/v3bt3fvWrXzVhouXPFEqWyiOPPJLDDjssXbt2zUknnZTnnnsuPXr0yE9/+tNMmDAhN998c9ERodFdeOGFSZJnnnkmv/nNb5Ik++yzT+6///6MGzcuxx9/fF588cUMGjSoyJhAMzd48OCMGTMmN910U95///0kyWmnnZZp06blmWeeyVZbbZUePXrkb3/7W8FJofHV1tZm1VVXXezxHTt2TG1tbRMmWv4UOBbbP//5z1x88cX57Gc/m5133jm//OUv06lTp5x11lmpq6vLTjvtlOOOO67Z3aoVPvH4449nr732ykYbbVS/7pNZ6O3bt8/VV1+drl275rvf/W5REVuMDz74IEOGDMlOO+2UjTbaKD179pzvtf766xcdE5rEfffdl5122imHHXZYKpVKg21bbrll/T8qnX/++QUlhKYza9asVFcv/iTCVq1aLfSSh7IyhZJFuvPOO3P99dfn97//febOnZuVV145hx9+eI444ojsuOOOqVQqueSSS4qOCU1u6tSp6dmzZ/1y69at8+GHH9YvV1VVpV+/fs3uYukVjeljtHQTJkzIAQccUL/cqlWr+v/vJ8mqq66aPfbYI7/+9a8zdOjQIiJCk/rvf7hoaRQ4FumAAw5IVVVVdt555xx++OHZd9990759+6JjwXLXpUuX+ulKSbLmmmvmtddeazBmxowZmT59+vKO1qL85/SxQw89NK1atcppp52W8847L88++2y++c1vprq6Or///e+LjgpNoqamJrNnz65fXnXVVfPWW281GNOhQ4e88847yzsaLBcXXnhhhgwZslhjm+NNxRQ4FktdXV3eeeedTJo0KR988IECR4u08cYb5+9//3v98nbbbZd77rknTz31VLbddtu8/PLL+fWvf51evXoVmLL5+8/pY//tk+ljm266ac4//3xnH2iWevbsmXHjxtUvb7bZZnnooYfy3nvvZfXVV89HH32U4cOHZ9111y0uJDSRddddt8WfgXMNHIv06KOP5rDDDstrr72W008/Peuss0723HPP/OpXv1rgbYuhudpzzz3z2GOPZcKECUlSf/3nF7/4xXTu3DmbbrpppkyZ4hq4JjZhwoRsttlm9cufNn0MmqNdd901I0aMqD/b//Wvfz3vvvtu+vTpkwMOOCCbbLJJxowZk4EDBxYbFJrAuHHjMnbs2CV+NScKHIv0pS99KTfeeGMmTJiQa665Jn379s3999+fQw45JGussUaOPfbYoiPCcnH88cfnn//8Z1ZfffUkSZ8+fTJixIjsvvvu6dSpU3beeecMHz48++yzT8FJmzfTx2jpjj/++PzsZz+rL3D77rtvLr300kybNi133nlnJk6cmNNPPz1nnHFGwUmBpuBB3iyVF198MT//+c/zi1/8Iu+9916Sj68HOv7443PkkUd6DhbQZLbddtusueaaufvuu5Mku+22W55//vn87W9/q58+1qdPn1RVVeWVV14pOC0sP3Pnzs3kyZPTpUuXFj/FDJozBY5lMmvWrNx99925/vrrM2LEiNTV1aVSqWSHHXbIiBEjio4HNEODBg3KD3/4w0ycODErrbRS7rrrruy///7p2rVrtt122/zlL3/JuHHjcvHFF+c73/lO0XEBaEQ33XTTUr3viCOOaOQkxVHgWKSePXvm1FNPzcknn/yp4958881cf/31ueGGG/LWW29l7ty5yykhNI3HHntsqd+7/fbbN2IS/tOECRPy2GOPZaeddkqnTp2SJJdffnkuuuiiTJ06Ne3bt88JJ5yQSy65JK1atSo4LTS+8ePH57XXXss222yTlVZaKUkyb968XHrppbn33nvTrl27nH766dlzzz0LTgqNr6qqarHOMH9yUiH5+LEDzelulAoci1RVVZXBgwfnvPPOW6zxdXV1efjhh7PLLrs0cTJoWov7l8SC+AeM5c/0MVqKgQMHZvjw4Zk4cWL98w4vvPDCDBo0qH5Mq1at8uSTT2bLLbcsKiY0iUsuuWSx/hs/Z86cDBs2LGPGjEmlUmlWfy97jACNrlKpKG80C+edd958f0k8/fTTefDBB7PBBhtku+22yxprrJF33nknTz75ZF599dXstttu2WabbQpK3DI89thj6dGjx3y3SG/VqlXWWGONJB+foRg7dqwzoTRLTzzxRHbeeef68lZXV5err746vXr1yu9///tMnDgxO++8cy699FJ3Y6XZWZyp8XfccUe+973vZcyYMampqclZZ521HJItPwocwEIMHjy4wfIf//jHDBkyJP/3f/+XY445pkG5q6ury89+9rOccsopOeecc5Zz0pZlxx13zKBBgz51VsBNN92U8847r1n9iyt84t13321ws7DRo0dn0qRJGTx4cNZZZ52ss846GTBgQB599NECU8Ly98c//jFnnnlmRo0alTZt2uS0007LOeeck9VWW63oaI3KYwRYLKYjQXLuuedmzz33zLHHHjvfn4lKpZL//d//zR577JFzzz23oIQtw+LM/J83b57/btFszZs3L/PmzatfHjlyZCqVSvr371+/bu21187EiROLiAfL3YsvvpivfOUr6devX5599tkcdthh+fvf/57LL7+82ZW3xBk4FtMPf/jD3HDDDYs9vlKpZMyYMU2YCJa/P//5zznllFM+dcxGG22UK6+8cjklYmFee+211NTUFB0DmsS6666bUaNG1S/fc889WWuttfK5z32uft3EiRPTsWPHAtLB8vPWW2/l3HPPzS233JK5c+dm9913zyWXXJLevXsXHa1JKXAslilTpmTKlClFx4BCtWnTJs8999ynjnnuuefSpk2b5ZSo5Tj66KMbLN9zzz0ZN27cfOPmzp2b8ePH57HHHssee+yxnNLB8rXffvvl4osvzv7775927drl8ccfz0knndRgzEsvvZSePXsWlBCa1pQpU3LxxRfnxz/+cWbMmJEtt9wyQ4cOTb9+/YqOtly4CyWLtKR3oYTm6uCDD86vf/3rXHzxxTn99NMbFLVZs2bl8ssvz/e+970ceOCB+eUvf1lg0uanqurfM/4rlcqnTqOsVCrZcsstc8stt+Szn/3s8ogHy1VtbW123XXX+rNwvXv3ziOPPJJVV101SfLGG2+kZ8+e+c53vpOLL764yKjQ6IYOHZqhQ4dm6tSpWX/99fP9738/+++/f9GxlisFjkVS4OBjb731VrbZZptMmDAhXbp0yRZbbJEuXbrk3XffzZ/+9Ke8++676dq1a5566qmss846RcdtVt54440kH1//9smzKRc0nbVVq1ZZddVV85nPfGZ5R4Tl7sUXX0zy8dTt/3zm4RtvvJHRo0dniy22yNprr11UPGgSnzziZ/3118/AgQNTXb14EwrPPPPMJk62/ChwLJICB/82ceLEfOc738mvf/3rzJgxo359u3bt8rWvfS2XXHJJ1lxzzQITNn833nhj+vbtmz59+hQdpcWaNWtW7rnnnjz77LOZMmXKAu/2WalUct111xWQDmjOlmRGxn+Oa053JVbgWCQFDuY3e/bs/P3vf8/UqVNTU1OTDTfc0LVvBaurq8s//vGPtGvXLt26dSs6TrP1xhtvZJdddsmYMWMWOZW1Of3ABKwYbrzxxqV635FHHtnISYrjJiYAS6F169bZZJNNio7RIt1111255557csUVV9Rf8zNu3Lh89atfzUsvvZQkOeCAA/KLX/yiwbQyGsdpp52Wf/zjHzn88MNz9NFHZ5111lnsKUw0ng8++CBXX311Hn744bz99tuZOXPmfGPcEZrmqDkVsaXlDByL9MYbb6Rjx45uyQ3/4Yknnsjo0aNTW1ubDh06pG/fvtluu+2KjtUi7LLLLnnnnXfywgsv1K/bZ5998pvf/Cb9+/fPe++9lxdeeCHXXnttjjvuuAKTNk8dO3bMlltumYceeqjoKC3WpEmT8oUvfCFjxoxJhw4dUltbm5qamsyaNSsfffRRkqRr165p3bp1xo4dW3BaoLF5kDeL1L1794WWt9mzZ+eKK67I3nvvnb333juXXXZZg+uCoLl58skn87nPfS7bb799Tj755Jxzzjk5+eSTs/3226dXr1556qmnio7Y7L300kvZaqut6pc/+OCD/O53v8uBBx6Yhx9+OKNGjcpGG22U66+/vsCUzde8efOy2WabFR2jRRs8eHDGjBmTm266Ke+//36Sj8+MTps2Lc8880y22mqr9OjRI3/7298KTgqN76abbmrwD3jJx9fl1tbWLnD8gw8+mNNPP315RFtuFDgW6aabbsq6666bESNGNFg/b9687LHHHjn99NMzfPjwDB8+PGeddVZ22GGHzJo1q6C00HT+9re/Zdddd81rr72WnXfeORdffHFuuOGGfP/7388uu+ySV199Nbvttlv9ND6axr/+9a8GN4p5/PHHM2fOnBx88MFJPp7e+sk1WjS+rbfeOi+//HLRMVq0++67LzvttFMOO+ywVCqVBtu23HLL3H///Rk3blzOP//8ghJC0xk4cGDuueeeBuuGDBlSP6X+vz399NO54oorlkOy5UeBY5EefPDBfPjhh/M9HPHmm2/OH/7wh3Tt2jXXX399fv3rX2fbbbfNn/70p1x77bXFhIUmdMEFF2TWrFm577778uCDD+Y73/lOjjzyyJx11ll54IEHct9992XGjBm54IILio7arHXo0CHvvfde/fIjjzySqqqqfOlLX6pf17p160ybNq2IeM3eJZdckj/84Q+54447io7SYk2YMKHBWdBWrVrVT51MklVXXTV77LFHfv3rXxcRD2hirjpmkf7yl79khx12mO9mALfccksqlUp+8YtfZPvtt0+SfPnLX0737t1z55135uSTTy4iLjSZkSNHZv/998/uu+++wO2777579t9///nOVtO4evXqleHDh+eiiy5Kq1at8stf/jL/8z//0+BfX994442sscYaBaZsvn73u99lxx13zIEHHpgddtghm2++eTp06DDfuEqlknPPPbeAhM1fTU1NZs+eXb+86qqr5q233mowpkOHDnnnnXeWdzRgOVDgWKR33nkne++9d4N18+bNq39Y8SflLUlWWmml7Lnnnvntb3+7vGNCk5s6dWrWW2+9Tx2z3nrrZerUqcspUct08skn54ADDsg666xTf6btoosuajDm6aefzuabb15QwuZt8ODB9b8eOXJkRo4cucBxClzT6dmzZ8aNG1e/vNlmm+Whhx7Ke++9l9VXXz0fffRRhg8fnnXXXbe4kECTUeBYpGnTps33rJ+XX34506dPz1577TXf+LXXXnuhF5JCmXXt2jVPP/30p4555pln0rVr1+WUqGXab7/98uMf/zg///nPU6lUctBBB2XgwIH12x999NHU1tYu9Ewpy+aRRx4pOkKLt+uuu+aHP/xhpk+fnpVWWilf//rXs//++6dPnz7Zdttt85e//CXjxo3LxRdfXHRUoAkocCxS586d89prrzVY98kPsVtsscV84z/66KOFXkgKZbbXXnvlqquuyrnnnptzzjkn7dq1q982Y8aMDBkyJI888ojpw8vBN77xjXzjG99Y4LYddtih/s58NL4ddtih6Agt3vHHH5+NN964vsDtu+++ufTSS3PRRRflzjvvTPv27XP66afnjDPOKDoq0AQ8B45FOuCAAzJ8+PCMHj06vXr1Sl1dXXbYYYc88cQTee6559K7d+8G43fddddMnjw5f/nLXwpKDE3jvffey9Zbb52xY8dm9dVXz1ZbbZU11lgj77zzTp599tlMmjQpPXv2zKhRo7LaaqsVHbdZmzNnTq666qrceuuteeWVVzJ9+vTMmTMnSTJ69Oj83//9X0499dRsuOGGBSdtnnz/K6a5c+dm8uTJ6dKly3x3p4TmoqqqKscee2yOPfbY+nU/+9nPcv311+fpp5+eb9bYJ9vmzp27vKM2GQWORXriiSey/fbbp0OHDtlxxx3z+uuv54UXXsgXv/jFPPbYYw3Gfvjhh+nSpUuOPPLIXHPNNQUlhqYzefLknHnmmbntttsaPPOwXbt2OfjggzN06NB06tSpwITN30cffZRdd901TzzxRDp37pzWrVtnwoQJ9X85T506NWuuuWa+9a1vzXdtHMvuk+//ySefTKdOnXz/BXjiiSdy55135swzz2zwSI1PTJgwIZdeemm+9rWvZZtttikgITSdqqqqBf4DRV1d3aeuV+BocYYNG5ZTTjklH3zwQZLkf/7nf3LnnXfOd4H0j3/843zzm9/MXXfdlQEDBhSQlP90zDHHZPjw4XniiSeywQYbFB2n9N5888107NgxHTp0yOzZs/PKK6+ktrY2HTp0SK9evdK6det88MEHef/99908oAmde+65ufjii3PJJZfkjDPOyPnnn58LL7ywwV/Ou+++e9577708++yzBSZtnnz/xdtvv/3ywgsvzHd5w3/acMMNs9lmm+VXv/rVckzWMlx99dW57LLLMmnSpOy222656qqrsvbaaxcdq8U46qijlup9N9xwQyMnKVAdLKbp06fXjRo1qm7MmDELHTN27Ni60aNH182cOXM5JmNhXn/99brWrVvXnXjiiUVHaRaqqqrqLrjggk8dc9FFF9VVVVUtp0Qt0wYbbFDXv3//+uXBgwfP951/4xvfqOvSpcvyjtYi+P6L161bt7qBAwd+6phjjjmmbt11111OiVqO22+/va5SqdRVVVXVde7cua5SqdR169at7rXXXis6Gi2IB3mz2Nq3b58tt9wyPXv2XOiYHj16pE+fPmnTps1yTNZynHDCCTnxxBMzadKkjB49OieccEJOPfXUXHHFFQu8acN6662XHXbYIQ8++GABaZufurq6+ebWL2gMTevNN99c4A2U/tMqq6zicQ5NxPdfvHfffXeRZ3zWXHPNvPvuu8spUctx2WWXpaamJi+++GLefffd3HPPPfnXv/6VnXbaKf/4xz+KjkcLocDxqQ455JDcddddhb2fhq699tpce+21ef/99zNmzJhce+21ufLKK3Paaadl4403XuB0pR49emT8+PEFpG2Z3nrrrayyyipFx2jWVllllUX+YDpmzJh07tx5OSVqWXz/xevYsWPefPPNTx3zxhtvZOWVV15OiVqOl19+OQceeGA22mijJB/fnfjuu+/OpEmTsvnmm+eAAw7I0Ucfnfvvvz8TJ07M0UcfnWOOOabg1M2Hn0s/5jECfKrbbrstvXr1yr777lvI+2nok+cvrbvuullttdXyyCOPZNasWXn88cczdOjQ7LTTThk+fHiD23w/99xzWWONNYqKXHoXXHBBg+WFPbR47ty5GT9+fG677TY3DWhi22yzTYYPH54pU6akY8eO820fP3587rvvvuyzzz7LP1wL4Psv3jbbbJO7774748ePT7du3ebb/uabb+aee+5J//79C0jXvNXV1c33//tddtklv/vd73LIIYfkzjvvTJL06tUrPXv2zLBhw1KpVHLdddcVkLb58XPpxxQ4Fmn06NG56aabio5BGj5/qV27dvXLu+yyS3bYYYfstdde2WOPPXLOOedk8803zx133JHnnnsuhx9+eFGRS2/w4MH1v65UKhk5cuRCS1zy8cO+hw4d2vTBWrAzzjgjO+64Y3baaadceeWV9bevnz59ep566ql885vfzJw5c3L66acXnLR58v0X7/TTT8/w4cOz3Xbb5aKLLsouu+yStdZaKxMmTMjvf//7fO9738tHH32Ub33rW0VHbXY23njjjBo1ar71O+64Y8aPH59XXnkl06ZNy7rrrpuOHTt68H0T8HOpu1CyCFVVH8+yXZrnyXzyf63BgwfnvPPOa9RcLNgzzzyTvffeO5MmTUry8TFYc80186c//Sldu3YtOF05Pfroo0k+/i779++fgQMH5sgjj5xvXKtWrbLaaqulV69e9X9uaDrXXHNNTjnllAXeFrpVq1b5yU9+0uAZQTQu33/xrrjiinzrW9+q/7u2UqnU/7qqqio/+tGPcuKJJxYZsVkaOnRozjnnnDz//PP5/Oc/X3ScFsfPpR9T4PhUN9544zLvo2/fvunTp08jpGFxvP/++/nZz36WMWPGZP3118/RRx/tuWSN5Pzzz8+OO+6Y7bffvugo5ONrUa699to888wz+de//pUOHTpk6623zgknnOAHq+XA91+8F154Iddee22effbZTJ06NR07dsxWW22V448/PptssknR8ZqlqVOn5vzzz88222yTr33ta0XHaXH8XPoxBQ4AAKAkzPMBAAAoCQUOAACgJBQ4ltrMmTMzePDgzJw5s+goLZZjUDzHoHiOQfEcg+I5BsVzDIrVkr5/18Cx1Gpra1NTU5OpU6emQ4cORcdpkRyD4jkGxXMMiucYFM8xKJ5jUKyW9P07AwcAAFASChwAAEBJVBcdoCWbN29e3n777ayyyipL9UDCotXW1jb4X5Y/x6B4jkHxHIPiOQbFcwyK5xgUqzl8/3V1dfnggw/StWvX+oeWL4hr4Ar01ltvpVu3bkXHAAAAVhDjx4/POuuss9DtzsAVaJVVVkmS7LDqoamualNwmpbrkIeeKzpCi3fDaXsXHaHFa/vOh0VHaNHmvjq26AjMm1t0Ako4G6nZcV6nUHMyO4/nvvqOsDAKXIE+mTZZXdVGgSvQSqu0KjpCi1dd3a7oCC1edavZRUdo0SqV1kVHoOK2AIVT4FYAClyh/v+vf1GXVvmvFQAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEArcI48aNS6VSycCBA4uOAgAAtHAKHAAAQElUFx1gRbf22mvn5ZdfTk1NTdFRAACAFk6BW4TWrVunV69eRccAAAAwhXJRFnQN3IQJE3LKKadkgw02SPv27dOxY8dstNFGOf744zN16tTiwgIAAM2aM3BLaPr06dluu+0ybty47Lrrrtlnn30ya9asjB07NjfffHO+/e1vm24JAAA0CQVuCY0YMSJjx47Nqaeemh/+8IcNtn344Ydp3br1Qt87c+bMzJw5s365tra2yXICAADNjymUS6l9+/bzrVt55ZXTtm3bhb5nyJAhqampqX9169atKSMCAADNjAK3hLbffvustdZaueSSS7LnnnvmmmuuyUsvvZS6urpFvvfss8/O1KlT61/jx49fDokBAIDmQoFbQjU1NXn66adzxBFH5Omnn84JJ5yQz3/+8+nevXt+8pOffOp727Ztmw4dOjR4AQAALC4Fbimsu+66GTZsWCZNmpTnnnsuQ4cOzbx583LiiSfm1ltvLToeAADQTClwy6Cqqip9+/bNmWeeWV/c7r333oJTAQAAzZUCt4T+9re/5Z133plv/Sfr2rVrt7wjAQAALYTHCCyhhx56KGeccUa22267bLjhhll99dXz+uuv59577027du1y4oknFh0RAABophS4JbTbbrtl3Lhxeeyxx3LXXXflww8/zNprr50DDzwwZ555ZjbeeOOiIwIAAM2UArcIPXr0aPCIgI022ig/+tGPigsEAAC0WK6BAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJKoLjoAybyptZlXaV10jBZr2FZ9i47Q4o146bqiI7R4u3xtYNERWrTqNv4OKNq8mfOKjkBdXdEJoBScgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQVuCYwbNy6VSiUDBw4sOgoAANACKXDLqF+/fqlUKkXHAAAAWoDqogOUydprr52XX345NTU1RUcBAABaIAVuCbRu3Tq9evUqOgYAANBCmUK5BP77GrhKpZJHH320/tefvFwjBwAANAVn4JbBoEGDMmzYsLzxxhsZNGhQ/fq+ffsWFwoAAGi2FLhlMHjw4IwcOTJvvPFGBg8evMjxM2fOzMyZM+uXa2trmzAdAADQ3JhCuRwNGTIkNTU19a9u3boVHQkAACgRBW45OvvsszN16tT61/jx44uOBAAAlIgplMtR27Zt07Zt26JjAAAAJeUMHAAAQEkocMuoVatWSZK5c+cWnAQAAGjuFLhltNpqqyWJ69kAAIAm5xq4ZdS/f//ccccd2W+//bLHHnukXbt26dOnT7761a8WHQ0AAGhmFLhldNxxx2XcuHG57bbbMnTo0MyZMydHHnmkAgcAADQ6BW4J9OjRI3V1dQ3WVVdXZ+jQoRk6dGhBqQAAgJbCNXAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEtVFByCpVLdKpeJQFGVu7YdFR2jxTn57y6IjtHhz27UqOkKLNveLny86QovX9qlXio7Q4s2bNq3oCFAKzsABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCtxDDhg1LpVLJsGHDGqzv0aNHevTosVhjAQAAGlOLLXDjxo1LpVLJwIEDi44CAACwWKqLDrCi2meffbLNNttkrbXWKjoKAABAEgVuoWpqalJTU1N0DAAAgHrNdgrlnXfemR122CFdunRJu3bt0rVr1+y888658847M2zYsKy33npJkhtvvDGVSqX+NXLkyCSuawMAAFY8zfIM3DXXXJMTTjgha621VvbZZ5+svvrqmThxYkaNGpW777473/72t3PKKafkiiuuSJ8+fTJgwID69/73DUoAAABWFM2ywP385z9PmzZtMnr06HTp0qXBtvfeey+rr756Tj311FxxxRXp27dvBg8evFxyzZw5MzNnzqxfrq2tXS6fCwAANA/Ndgpl69at07p16/nWr7766gWk+diQIUPqr62rqalJt27dCssCAACUT7MscAcddFCmTZuWTTbZJGeccUbuu+++FeJs19lnn52pU6fWv8aPH190JAAAoESaZYH79re/neuuuy5du3bN5Zdfnj333DOrr756BgwYkLFjxxaWq23btunQoUODFwAAwOJqlgWuUqnk6KOPzrPPPptJkybl7rvvzr777pvf/OY3+cpXvpK5c+cWHREAAGCJNcubmPynT868DRgwIJMnT84f/vCH/OMf/8hKK62UJMocAABQGs3yDNzIkSNTV1fXYN3s2bPzr3/9K0nSrl27rLrqqqlUKq5DAwAASqNZnoEbMGBAOnTokG222Sbdu3fP7Nmz89BDD+Wll17K/vvvn+7duydJttxyyzz22GM5/PDDs8EGG6SqqiqHH354/XYAAIAVSbMscEOGDMkDDzyQUaNGZfjw4fnMZz6T9ddfP9dcc02OOeaY+nE333xzTjvttPz2t7/N1KlTU1dXly9+8YsKHAAAsEKq1P33XEOWm9ra2tTU1KR/u6+lutKm6Dgt1rxZs4uO0OJ9blSznM1dKi9+p3fREaBQbZ96pegILd68adOKjgCFmlM3OyPzm0ydOvVT71bvpyYAAICSWOIC179//0yZMmW+9bW1tenfv39jZAIAAGABlrjAjRw5MrNmzZpv/YwZM/LHP/6xUUIBAAAwv8W+ickLL7xQ/+uXXnopEydOrF+eO3duHnjggay99tqNmw4AAIB6i13g+vbtm0qlkkqlssCpku3bt89VV13VqOEAAAD4t8UucGPHjk1dXV169uyZUaNGpXPnzvXb2rRpky5duqRVq1ZNEhIAAIAlKHCfPBtt3rx5Cx1TV1eXSqWy7KkAAACYzxLfxGTgwIGZtoDndIwbNy7bb799o4QCAABgfktc4J5//vn07t07Tz31VP26G2+8MX369EmnTp0aNRwAAAD/tthTKD8xatSofPe7302/fv3yrW99K//4xz9y//335//9v/+X4447rikyAgAAkKUocK1bt86ll16alVZaKRdeeGGqq6vz6KOPZtttt22KfAAAAPz/lngK5ezZs/Otb30rQ4cOzdlnn51tt902++67b+67776myAcAAMD/b4nPwG2xxRaZPn16Ro4cmW222SZ1dXX5wQ9+kH333TdHH310fvKTnzRFTgAAgBZvic/AbbHFFhk9enS22WabJEmlUslZZ52Vp556Ko899lijBwQAAOBjS3wG7rrrrlvg+s022yx//vOflzkQAAAAC7bEZ+CS5Oabb852222Xrl275o033kiS/OhHP8oDDzzQqOEAAAD4tyUucNdcc01OP/30fPnLX86UKVMyd+7cJEnHjh3zox/9qLHzAQAA8P9b4gJ31VVX5Wc/+1nOOeectGrVqn79Fltskb/+9a+NGg4AAIB/W+ICN3bs2Gy22WbzrW/btm2mTZvWKKEAAACY3xIXuPXWWy+jR4+eb/0DDzyQjTbaqDEyAQAAsABLfBfK008/PSeeeGJmzJiRurq6jBo1KrfeemuGDBmSn//8502REQAAgCxFgTv22GPTvn37fO9738v06dNzyCGHpGvXrrniiity0EEHNUVGAAAAshQFLkkOPfTQHHrooZk+fXo+/PDDdOnSpbFzAQAA8F+W+Bq4/v37Z8qUKUmSlVZaqb681dbWpn///o0aDgAAgH9b4gI3cuTIzJo1a771M2bMyB//+MdGCQUAAMD8FnsK5QsvvFD/65deeikTJ06sX547d24eeOCBrL322o2bDgAAgHqLXeD69u2bSqWSSqWywKmS7du3z1VXXdWo4QAAAPi3xS5wY8eOTV1dXXr27JlRo0alc+fO9dvatGmTLl26pFWrVk0SEgAAgCUocN27d0+SzJs3r8nCAAAAsHBLfBMTAAAAiqHAAQAAlIQCBwAAUBIKHAAAQEkscYHr2bNn3nvvvfnWT5kyJT179myUUAAAAMxviQvcuHHjMnfu3PnWz5w5M//85z8bJRQAAADzW+zHCNx77731v37wwQdTU1NTvzx37tyMGDEiPXr0aNRwAAAA/NtiF7gBAwYkSSqVSo488sgG21q3bp0ePXrk8ssvb9RwAAAA/NtiF7hPHuC93nrr5dlnn02nTp2aLBQAAADzW+wC94mxY8c2RQ4AAAAWYYkL3AUXXPCp288777ylDtNSVdq1TaXSpugYLdfMmUUnaPH+cXD3oiO0eK3WnFd0hBbt9f8tOgFrf+bzRUdo8VZ55o2iI7R482o/KDpCi1ZVNyuZvuhxS1zg7r777gbLs2fPztixY1NdXZ31119fgQMAAGgiS1zgnnvuufnW1dbWZuDAgdlnn30aJRQAAADzW+LnwC1Ihw4dcv755+fcc89tjN0BAACwAI1S4JJk6tSpmTp1amPtDgAAgP+yxFMor7zyygbLdXV1mTBhQm6++ebssccejRYMAACAhpa4wP3whz9ssFxVVZXOnTvnyCOPzNlnn91owQAAAGjIc+AAAABKYpmugRs/fnzGjx/fWFkAAAD4FEtc4ObMmZNzzz03NTU16dGjR3r06JGampp873vfy+zZs5siIwAAAFmKKZTf/OY3c9ddd+UHP/hBtt122yTJU089lcGDB+e9997LNddc0+ghAQAAWIoC98tf/jK33XZbgztO9u7dO926dcvBBx+swAEAADSRJZ5C2bZt2/To0WO+9eutt17atGnTGJkAAABYgCUucCeddFIuvPDCzJw5s37dzJkzc/HFF+ekk05q1HAAAAD82xJPoXzuuecyYsSIrLPOOunTp0+S5Pnnn8+sWbOy0047Zd99960fe9dddzVeUgAAgBZuiQtcx44ds99++zVY161bt0YLBAAAwIItcYG74YYbmiIHAAAAi7DE18D1798/U6ZMmW99bW1t+vfv3xiZAAAAWIAlLnAjR47MrFmz5ls/Y8aM/PGPf2yUUAAAAMxvsadQvvDCC/W/fumllzJx4sT65blz5+aBBx7I2muv3bjpAAAAqLfYBa5v376pVCqpVCoLnCrZvn37XHXVVY0aDgAAgH9b7AI3duzY1NXVpWfPnhk1alQ6d+5cv61Nmzbp0qVLWrVq1SQhAQAAWIIC17179yTJvHnzmiwMAAAAC7fEjxG46aabPnX7EUccsdRhAAAAWLglLnCnnHJKg+XZs2dn+vTpadOmTVZaaSUFDgAAoIks8WME3n///QavDz/8MH//+9/zxS9+MbfeemtTZAQAACBLUeAWZIMNNsgll1wy39k5AAAAGk+jFLgkqa6uzttvv91YuwMAAOC/LPE1cPfee2+D5bq6ukyYMCFXX311tttuu0YLBgAAQENLXOAGDBjQYLlSqaRz587p379/Lr/88sbKBQAAwH9Z4gLnOXAAAADFWOpr4CZPnpzJkyc3ZhYAAAA+xRIVuClTpuTEE09Mp06dssYaa2SNNdZIp06dctJJJ2XKlClNFBEAAIBkCaZQ/utf/8q2226bf/7znzn00EOz0UYbJUleeumlDBs2LCNGjMiTTz6ZVVddtcnCAgAAtGSLXeAuuOCCtGnTJmPGjMkaa6wx37Zdd901F1xwQX74wx82ekgAAACWYArlPffck8suu2y+8pYka665Zn7wgx/k7rvvbtRwAAAA/NtiF7gJEybk85///EK3b7LJJpk4cWKjhAIAAGB+i13gOnXqlHHjxi10+9ixY7Paaqs1RiYAAAAWYLEL3G677ZZzzjkns2bNmm/bzJkzc+6552b33Xdv1HAAAAD82xLdxGSLLbbIBhtskBNPPDG9evVKXV1dXn755fzkJz/JzJkzc/PNNzdlVgAAgBZtsQvcOuusk6eeeionnHBCzj777NTV1SVJKpVKdtlll1x99dXp1q1bkwUFAABo6Ra7wCXJeuutl/vvvz/vv/9+XnvttSTJZz/7Wde+AQAALAeLfQ3cf1p11VWz1VZbZauttmqU8jZy5MhUKpUMHjx4mfcFAADQXC1VgQMAAGD5U+AAAABKQoEDAAAoiRWuwP3pT3/KLrvsklVWWSU1NTXZZ599FvgA8SeeeCJ77rlnVltttbRr1y69evXKoEGDMn369PnGViqV9OvXL//85z9zyCGHpFOnTllllVWy55575vXXX0+SvPzyyxkwYEBWW221rLLKKtl///3zzjvvLDDjCy+8kIMOOihrrbVW2rRpk+7du+eb3/xm3nvvvUb9LgAAAP7TClXgnn322Wy//fZp06ZNvv71r2eLLbbIPffck5133jkzZsyoH3f77bdnhx12yMiRIzNgwICceuqpWWmllXLBBRekf//+DcZ+4v33388Xv/jFjB07NkceeWT69euX++67L7vssktefPHFfOELX8iHH36Yo48+OltssUXuvPPOHHzwwfPt5957781WW22Ve++9N/369cupp56aTTfdNFdffXW23XbbvP/++wv9/c2cOTO1tbUNXgAAAItriR4j0NTuu+++3HbbbTnwwAPr1x1xxBG5+eabc8899+Sggw5KbW1tjjvuuFRXV+epp55K7969kyTf//73c8ghh+RXv/pVLr300px77rkN9v3CCy/ktNNOy//7f/+vft0JJ5yQa665Jl/60pcyePDgnHLKKUmSurq6fOUrX8l9992Xv/zlL9l8882TJO+9914OP/zwdOrUKU888US6d+9ev6/bbrstBx98cM4777xcddVVC/z9DRkyJOeff37jfFkAAECLs0Kdgdt+++0blLckOfroo5N8fHYuSX7zm99k6tSpOfroo+vLW5JUVVXlBz/4QaqrqzNs2LD59r3yyivnoosuarDukzNsq6++ek4++eT69ZVKJQcddFCS5Pnnn69ff9NNN6W2tjZDhgxpUN6S5KCDDsrmm2+e2267baG/v7PPPjtTp06tf40fP36hYwEAAP7bCnUG7n/+53/mW7fOOuskSaZMmZIkee6555Ik/fr1m2/suuuum549e+bVV1/NBx98kFVWWaV+2wYbbJCVVlqpwfi11lorSdK7d+9UKpUFbnv77bfr1z399NNJkmeeeSZjxoyZ7/NnzJiRyZMnZ/LkyenUqdN829u2bZu2bdvOtx4AAGBxrFAFrkOHDvOtq67+OOLcuXOTpP66sTXWWGOB+1hrrbXy6quvpra2tkGB+7R9f9q22bNn16/717/+lST58Y9//Km/j2nTpi2wwAEAACyLFWoK5eL4pGwt7A6REydObDCuKT77r3/9a+rq6hb6+u/plQAAAI2hdAVus802S5KMHDlyvm3jx4/PmDFj0rNnzwZn3xrL1ltvnSR56qmnGn3fAAAAi1K6Arf33nunpqYmN9xwQ/72t7/Vr6+rq8tZZ52VOXPmZODAgU3y2UcddVRWWWWVnHPOOQ0++xPTp0+vv04OAACgsa1Q18Atjg4dOuRnP/tZDj744Gy99dY58MAD07lz5zz88MP585//nK222ipnnHFGk3x2586dc+utt+aAAw5Inz59svvuu6dXr16ZOXNmxo0bl0cffTRf+MIX8sADDzTJ5wMAAC1b6QpckhxwwAFZc801M2TIkNx1112ZPn16evTokXPPPTdnnXVW2rVr12Sfveeee+a5557LpZdemocffjgPPfRQPvOZz2SdddbJUUcdlcMOO6zJPhsAAGjZKnV1dXVFh2ipamtrU1NTk506Hp7qSpui47RYc6fWFh2hxWv12fWKjtDizV6zpugILdrr/1t0Ata+o3XREVq8VZ55o+gILd682g+KjtCizamblT9Mvy1Tp0791Bsylu4aOAAAgJZKgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkqosOQJLVVk1atS06Rcs1tbboBC3e3H+MLTpCi9d6cseiI7Ro6/x6g6IjtHgrveHvgqK9NLh70RFavM/eOrvoCC3anDkzkscXPc4ZOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoFbClOmTMnQoUOz9dZbZ5VVVkmHDh2y9dZb5/bbby86GgAA0IwpcEvh/PPPz9lnn5327dvnG9/4Rg466KC88sor+drXvpYrr7yy6HgAAEAzVV10gDLaeuut8/zzz2fTTTetX3fKKadk0003zWWXXZaTTz65wHQAAEBzpcAthYMOOmi+dZ///Oez+uqrZ9KkSQUkAgAAWgIFrpHce++9mTx5cvbaa6+Fjpk5c2ZmzpxZv1xbW7s8ogEAAM2Ea+AawUMPPZSDDz44Xbt2zdVXX73QcUOGDElNTU39q1u3bssxJQAAUHYK3DJ65JFHstdee2XVVVfNH/7wh08tZWeffXamTp1a/xo/fvxyTAoAAJSdKZTLYPbs2TnkkEPSqlWrjBgxIp/73Oc+dXzbtm3Ttm3b5ZQOAABobpyBWwavvPJKJk6cmN12222R5Q0AAGBZKXDLYNq0aUmSVVZZpeAkAABAS2AK5TJYd911M2TIkAbPgwMAAGgqCtwy6NixYwYMGJCampqiowAAAC2AKZTLYNSoUdloo41y9tlnFx0FAABoARQ4AACAkjCFchn069cvdXV1RccAAABaCGfgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCSqiw5AUvf2xNRV2hQdo8WqVLcuOkKLVzd7VtERWry5U2uLjtCitfvts0VHaPHmFR2ArHfH5kVHaPF++8v/KzpCi1b7wbys+blFj3MGDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkFDgAAICSUOAAAABKQoEDAAAoCQUOAACgJBQ4AACAklDgAAAASkKBAwAAKAkFDgAAoCQUOAAAgJJQ4AAAAEpCgQMAACgJBQ4AAKAkCi1ww4YNS6VSqX8ddNBBRcZZYjNmzGiQv1KpFB0JAABoxlaIM3B77713Bg0alP33379+3cCBA+crR//9GjZsWP34/yyDQ4YMWeDnXHLJJfO9L0l69OjRYL9t27ZN586ds9VWW+XEE0/M448/vsD9VVdXZ9CgQRk0aFC6d+++zN8DAADAp6kuOkCSDBgwIAMHDlzgtmOOOSbrrLPOArf17dt3geuHDh2ar3/961lttdUWO0OrVq3yve99L0kyZ86cvP/++/nrX/+an/70p/nJT36Sr371q7nxxhuz6qqr1r+nuro6gwcPTpKMHDkyb7zxxmJ/HgAAwJJaIQrcpzn22GOzzTbbLPb49ddfP2PGjMnFF1+cyy+/fLHf959l7D+98cYbOeaYYzJ8+PDss88++cMf/pCqqhXixCUAANDCNLsmMnDgwHz2s5/Nj3/847z55pvLvL/u3btn+PDh2WijjfLoo4/mjjvuaISUAAAAS67ZFbjq6upcfPHFmTlzZs4999xG2Wf79u3z7W9/O0nyq1/9aqn3M3PmzNTW1jZ4AQAALK4Vfgrlz3/+8zzwwAML3Pad73wn7dq1m2/9AQcckMsuuyy33HJLvvWtb6V3797LnKNfv35JkmeffXap9zFkyJCcf/75y5wFAABomVb4AnfdddctdNupp566wAJXqVQydOjQ9O/fP9/5zndy3333LXOOrl27JkkmT5681Ps4++yzc/rpp9cv19bWplu3bsucDQAAaBlW+CmUTz31VOrq6hb46tix40Lft+OOO2b33XfP/fffn0cffXT5Bf4Ubdu2TYcOHRq8AAAAFtcKX+CWxSWXXJKqqqqceeaZy7yvt99+O0nSuXPnZd4XAADA0mjWBa5Pnz459NBDM2rUqNx+++3LtK+RI0cmSbbccstGSAYAALDkmnWBS5ILL7wwbdu2zTnnnJM5c+Ys1T4++uij+mfKHXzwwY0ZDwAAYLE1+wLXvXv3nHDCCXnttdcybNiwJX7/m2++ma9+9at56aWXsuOOO2bfffdt/JAAAACLYYW/C+WnPUZgm222ye67777IfZxzzjm5/vrrM2bMmIWOmTNnTgYPHpwkmTt3bqZMmZIXXnghTzzxRObOnZu99947w4YNS6VSWarfBwAAwLJa4Qvcpz1G4JRTTlmsArf66qvnrLPOyne/+92Fjpk7d279M9ratGmTDh06ZL311svXv/71HHLIIdluu+2WPDwAAEAjWmEL3LBhw5ZoyuPAgQMzcODAhW4/++yzc/bZZy9w27hx45YsHAAAQAFWiGvgjjrqqFQqlRx00EFFR1kiM2bMSKVSSaVSWWGeNQcAADRfhZ6B69u3bwYNGlS/vMkmmxSYZslVV1c3yA8AANCUCi9wffv2LTLCMqmurq6/8QkAAEBTWyGmUAIAALBoChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACWhwAEAAJSEAgcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACVRXXSAlqyuri5JMqdudsFJWrZPjgPFqfNnoHh184pO0LL5/iFz5swoOkKLV/uB/xYV6YMPP/7+F/WzaaXOT6+Feeutt9KtW7eiYwAAACuI8ePHZ5111lnodgWuQPPmzcvbb7+dVVZZJZVKpeg4S6y2tjbdunXL+PHj06FDh6LjtEiOQfEcg+I5BsVzDIrnGBTPMShWc/j+6+rq8sEHH6Rr166pqlr4lW6mUBaoqqrqU9t1WXTo0KG0f1CaC8egeI5B8RyD4jkGxXMMiucYFKvs339NTc0ix7iJCQAAQEkocAAAACWhwLHU2rZtm0GDBqVt27ZFR2mxHIPiOQbFcwyK5xgUzzEonmNQrJb0/buJCQAAQEk4AwcAAFASChwAAEBJKHAAAAAlocABAACUhAIHAABQEgocAC3ewIEDM2DAgOX+ucOGDUvHjh0XOW7w4MHp27dvo39+U+0XgKajwAEAAJSEAgcA/6Vfv345+eSTc+aZZ2a11VbLmmuumcGDBzcYU6lUcs0112SPPfZI+/bt07Nnz9xxxx3120eOHJlKpZIpU6bUrxs9enQqlUrGjRuXkSNH5qijjsrUqVNTqVRSqVTm+4zk47N0559/fp5//vn6ccOGDUuSTJkyJccee2w6d+6cDh06pH///nn++eeTJJMmTcqaa66Z73//+/X7evLJJ9OmTZuMGDHiU/cLwIqruugAALAiuvHGG3P66afnmWeeyVNPPZWBAwdmu+22yy677FI/5txzz80ll1ySK664IjfffHMOOuig/PWvf81GG220yP1/4QtfyI9+9KOcd955+fvf/54kWXnllecbd+CBB+bFF1/MAw88kIcffjhJUlNTkyQ54IAD0r59+9x///2pqanJT3/60+y000559dVX07lz51x//fUZMGBAdt1113zuc5/L4YcfnpNOOik77bRTPvroo4XuF4AVlwIHAAvQu3fvDBo0KEmywQYb5Oqrr86IESMaFLgDDjggxx57bJLkwgsvzEMPPZSrrroqP/nJTxa5/zZt2qSmpiaVSiVrrrnmQse1b98+K6+8cqqrqxuMe/zxxzNq1Ki8++67adu2bZLksssuyz333JM77rgj//u//5svf/nLOe6443LooYdmiy22yGc+85kMGTLkU/cLwIpNgQOABejdu3eD5bXWWivvvvtug3XbbrvtfMujR49u6mhJkueffz4ffvhhVl999QbrP/roo4wZM6Z++bLLLssmm2yS22+/PX/+85/ryx4A5aTAAcACtG7dusFypVLJvHnzFvv9VVUfX2ZeV1dXv2727NmNEy7Jhx9+mLXWWisjR46cb9t/3tlyzJgxefvttzNv3ryMGzcum266aaNlAGD5U+AAYCk9/fTTOeKIIxosb7bZZkmSzp07J0kmTJiQVVddNUnmOzvXpk2bzJ07d5Gfs6Bxm2++eSZOnJjq6ur06NFjge+bNWtWDjvssBx44IH53Oc+l2OPPTZ//etf06VLlyX6fABWHO5CCQBL6fbbb8/111+fV199NYMGDcqoUaNy0kknJUk++9nPplu3bhk8eHBee+21/O53v8vll1/e4P09evTIhx9+mBEjRmTy5MmZPn36Aj+nR48eGTt2bEaPHp3Jkydn5syZ2XnnnbPttttmwIAB+f3vf59x48blySefzDnnnJM//elPSZJzzjknU6dOzZVXXpmzzjorG264YY4++uhP3S8AKzYFDgCW0vnnn5/bbrstvXv3zk033ZRbb701G2+8cZKPp2DeeuuteeWVV9K7d+8MHTo0F110UYP3f+ELX8jxxx+fAw88MJ07d84PfvCDBX7Ofvvtl9133z077rhjOnfunFtvvTWVSiX33Xdftt9++xx11FHZcMMNc9BBB+WNN97IGmuskZEjR+ZHP/pRbr755nTo0CFVVVW5+eab88c//jHXXHPNQvcLwIqtUvefk/MBgMVSqVRy9913Z8CAAUVHAaAFcQYOAACgJBQ4AACAknAXSgBYCq5AAKAIzsABAACUhAIHAABQEgocAABASShwAAAAJaHAAQAAlIQCBwAAUBIKHAAAQEkocAAAACXx/wEH1VU/We3aDwAAAABJRU5ErkJggg==",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"model.plot_attention('¿Todavía está en casa?') # Are you still home"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JHBdOf9duumm"
},
"source": [
"Translate a few more sentences and plot them:"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:10.812417Z",
"iopub.status.busy": "2023-12-07T12:24:10.812113Z",
"iopub.status.idle": "2023-12-07T12:24:11.211255Z",
"shell.execute_reply": "2023-12-07T12:24:11.210549Z"
},
"id": "flT0VlQZK11s"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmpfs/tmp/ipykernel_16852/3355722706.py:23: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_xticklabels([''] + context, fontdict=fontdict, rotation=90)\n",
"/tmpfs/tmp/ipykernel_16852/3355722706.py:24: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_yticklabels([''] + output, fontdict=fontdict)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 239 ms, sys: 47 ms, total: 286 ms\n",
"Wall time: 198 ms\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3AAAAMZCAYAAABBEdZFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFy0lEQVR4nO3deZzVdd3//+cZBkZMAUXcEdylDLHUNFMBs9QycftqLoilX0vcLrtcuEwFzUTT3EpaXEBN6ptbai4VCS5hWu4/tRRBIdRcAkRln98fXU4i6wwDh/fM/X67nVucz/mcM6/xkziP+WyV+vr6+gAAALDSq6n2AAAAACwdAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFAIAQcAAFCI2moPQLn69u27zJ8xYMCA9O/fvxmmAQCAlk/A0WSjR49epvdXKpX07t27WWYBAIDWwCGULJPBgwdn3rx5TXrU19dXe3wAACiKgAMAACiEQyhpsnvuuSebbbZZ1d4PAACtTaXecWwAAABFcAgly2STTTbJFVdcUe0xAACgVRBwLJMJEyZkypQp1R4DAABaBQEHAABQCAEHAABQCAHHMqtUKtUeAQAAWgVXoWSZ1NTUpFOnTunUqdNSv6dSqWTcuHHLbygAAGihBBzLpKamaTtx582b18yTAABAy+cQSpbZ4MGDM2/evEY9AACAxhNwAAAAhRBwAAAAhRBwAAAAhRBwrFDz5s3L8OHDqz0GAAAUyVUoWSavvPJKOnXqlI4dOy5x3V/84hc599xz89JLL2Xu3LkrYDoAAGhZ7IFjmXTr1i1z5szJkCFD8rWvfS37779/Lr300nzwwQcN69xxxx351Kc+lf79++ell17K/vvvX8WJAQBYWbVp02aZH+eee261v43lqrbaA1C2N954IzvssEMmTZqUD3fm/uY3v8ltt92WUaNG5eijj86NN96YJOnXr1/OOeec9OzZs5ojAwCwkqqvr0+3bt3SvXv3Jr33gQceaP6hVjICjmVy/vnnZ+LEidl7770zYMCAJMm1116b++67L7vvvnseeuih7Lrrrrn88suzzTbbVHdYAABWekcddVTOPvvsJr23pqblH2Ao4Fgm9913Xz75yU/mrrvualh2wAEHZOutt87DDz+c/v37u2gJAAA0EwHHMpk4cWKOOeaY+ZZVKpV88YtfzAsvvJDBgwdXZzAAAIrzwQcfpLa26YmyrO8vQcv+7ljuZsyYkc6dOy+wfM0110ySJh2/DABA61RXV1fV95eg5R8kCgAA0ELYA8cye+ihh3LRRRctsCxJfvCDH2Rhtxo87bTTVshsAACU6+abb86vfvWrPPbYY3nrrbeSJF26dMn222+fQw89NP369avugFXgRt4sk6Zc6adSqbiRNwAAizRlypTsv//+GTNmzEJ3BiT//pmyb9++ueWWW9KhQ4cVPGH12APHMrnuuuuqPQIAAC3MkUcemdGjR6dHjx757//+7/Tu3Tvrr79+kmTy5Mm5//77c8kll2TUqFE56qijcsstt1R54hXHHjgAAGCl8fDDD2eXXXbJF7/4xdxxxx1ZZZVVFrrejBkz8tWvfjX3339/Hnrooey0004reNLqcBETVqhZs2bll7/8ZbXHAABgJTVy5MjU1tbmmmuuWWS8Jckqq6ySa6+9NjU1NRk5cuQKnLC67IFjhXjqqadyzTXX5Be/+EWmTJniHDgAABbqc5/7XNq3b5/Ro0cv1fq9e/fOjBkz8sgjjyzfwVYS9sCx3EydOjXDhg3Ldtttl8985jP50Y9+lBkzZuTQQw+t9mgAAKykxo8fn09/+tNLvX7Pnj0zfvz45TjRysVFTGh2999/f6655prcdtttmTFjRpJ/39D7jDPOyMEHH9yqrhIEQMsyadKk3H///Zk8eXJmzpy5wOuVSiVnnXVWFSaDlmPatGlZY401lnr9Tp06Zdq0actxopWLgKNZ/OMf/8jw4cNz3XXXZfz48amvr0/Xrl1z2GGHZejQodl9991zzDHHVHtMFuHdd9/Nj370o/zhD39Y7A8l48aNq8J0ACuHU089NZdffvl8pwHU19enUqnM92cBB8tm1qxZqa1d+kxp06ZNZs2atRwnWrkIOJbJLbfckmuvvTa/+93vMnfu3Ky22mo54ogj0r9///Tp0yeVSiVDhw6t9pgsxptvvpnPf/7zGTduXDp06JBp06alY8eOmTVrVj744IMkyfrrr5+2bdtWeVKA6vn5z3+eSy65JHvssUe+9a1v5YADDsiAAQPy5S9/OQ888ECuvvrq9OvXL8cdd1y1R4UW4cNfjLAgFzFhmdTU1KSmpiZf/OIXc8QRR2T//fdP+/btF1jn6KOPzs9+9rMqTcniDBw4MMOGDcv111+fww47LG3atMngwYNz9tln57HHHssJJ5yQ2tra/O53v8uqq65a7XEBqmL77bfP22+/nZdeeqnhv30f/l2ZJGPGjMkee+yRm266KQceeGCVp4Wy1dTUpLa2dqn3ws2ZMydz585tNRfJsweOZVZfX5833ngjb775Zt59990FAo6V2913353dd989hx9++AKvbb/99rnnnnvy6U9/OkOGDMmFF15YhQkBqu+FF17IEUcckZqa/1z/bc6cOQ1/3m233fKVr3wlF198sYCDZbTRRhvZA7cYAo5lMmbMmFx99dW55ZZbcsopp+S0007LHnvskf79+6dfv36pq6ur9ogswWuvvZaDDjqo4XmbNm0aDp1MkjXWWCN77bVX/t//+38CDmjVOnXq1PDnT3ziE3n77bfne33LLbfMH/7whxU8FbQ8EyZMqPYIKzW3EWCZ7LLLLhkxYkRee+21DBs2LL169co999yTQw89NOuss06OPvroao/IEnTs2DGzZ89ueL7GGmtk0qRJ863ToUOHvPHGGyt6NICVxgYbbDDf342bbrpp/vznP8+3zrPPPptPfOITK3o0oJURcDSL1VdfPccee2weffTRPP300znhhBPStm3bXHvttUmSu+66K+eee25eeeWVKk/Kx22yySbz/aZr2223ze9///uG3yx/8MEHufPOO7PRRhtVaUKA6tt5553nu0nwvvvumyeeeCLHHntsfvvb32bQoEG55557suuuu1ZxSqA1cBETlptZs2bltttuy7XXXptRo0Y1XF55t912y6hRo6o9Hv/rnHPOyaWXXprXX389q666am699dYceOCBWX/99bPTTjvl8ccfz4QJE3L++efnjDPOqPa4AFUxevToXHjhhfnJT36Sbt26Zfr06dltt93yxBNPpFKppL6+Pt27d8/999+fbt26VXtcKNr111/fpPf179+/mSdZOQk4lskmm2ySk08+OSeeeOJi13v11Vdz7bXX5rrrrsukSZNazVWCSvDaa6/lgQceyO6775611lorSXLJJZfke9/7XqZOnZr27dvnuOOOy9ChQ9OmTZsqTwuw8pg9e3Z+85vfZNy4cenWrVv22Wcfh1BCM6ipqVmqi5h89D6MlUplvgsLtWQCjmXy8csoL0l9fX3+8Ic/ZI899ljOk7Gs5s6dm7feeitrr722K0EBACvM0KFDl+pnjzlz5mT48OEZN25cKpVKq9lB4CqUrFCVSkW8rWQeeOCBdO/efYFz3Nq0aZN11lknSTJx4sSMHz/euR0AwHK3NKds3Hzzzfnud7+bcePGpWPHjjn99NNXwGQrBwEHrVyfPn1yzjnnLHYv6vXXX5+zzz671fxmC+Ab3/hGk95XqVRyzTXXNPM0wIcefPDBnHbaaXn00UfTrl27/Nd//VfOPPPMrLnmmtUebYURcCwzh9eVbWmOop43b57tDLQqw4cPX+jyDy9YsqjlAg6Wj2effTZnnHFG7rnnnlQqlRx++OE577zzWuVVsgUcy+zSSy/Nddddt9TrVyqVjBs3bjlORHN78cUX07Fjx2qPAbDCjB8/fr7n8+bNy0knnZRHHnkkJ510UnbZZZess846eeONN/LAAw/kiiuuyE477ZRLL720ShNDyzRp0qScddZZufHGGzN37tzsueeeGTp0aHr27Fnt0apGwLHMpkyZkilTplR7DBrh44cG3X777fPdC+5Dc+fOzcSJE/PAAw9kr732WkHTAVTfx28FMHTo0Pz5z3/OU089lfXWW69h+ZZbbpldd901Rx11VLbddtvcfPPNOe2001b0uNDiTJkyJeeff35+/OMfZ8aMGdl+++1z4YUXpnfv3tUerepchZJl0tirULJyqKmpafjzog4H+ujr22+/fW688cZsttlmK2I8gJXO5ptvnj333DNXXnnlItc5/vjjc9999+XFF19cgZNBy3PhhRfmwgsvzNSpU7Ppppvm+9//fg488MBqj7XSsAcOWqEPDw2qr69vuJffSSedtMB6bdq0yRprrOG+RkCrN2nSpKyyyiqLXWeVVVbJpEmTVtBE0HINGjQolUolm266aQYMGJCXX345F1100RLf11r2ftsDxzKxB658I0aMSK9evbLNNttUexRosc4999xUKpUMHDgwa665Zs4999ylel+lUslZZ521nKdjaWy++eapr6/Ps88+u9CQe//99/PpT386NTU19sDBMmrMkUIfXa+1XC1bwLFMBFzLVV9fn5deeimrrLJKunbtWu1xWEq228qppqYmlUolzz//fLbYYov5fjhZnNb0A8nK7sILL8ygQYOy7bbb5uyzz84XvvCFdO7cOW+//XYefPDBnHvuuXnqqadywQUXtJq9ALC8jBgxoknvO/LII5t5kpWTQyihlbv11ltz++235/LLL88aa6yRJJkwYUL22WefPPfcc0mSgw46KL/4xS/Spk2bao7KR9huZbn//vuTpOFy1x8+pxynnnpq/v73v+e6667L/vvvn+TfYT5v3rwk//7lyVFHHZVTTz21mmNCi9BaQqyp7IFjmbzyyivp1KmTS8wXbI899sgbb7yRp59+umHZfvvtl9/85jfp27dv3n777Tz99NP5yU9+kmOOOaaKk/JRthtUx5gxYzJixIg8/fTTmTp1ajp27JhtttkmRxxxhKvjASuEgGO5mT17dq666qr88Y9/TJLssssuOf7445d4Ejgr1gYbbJC99torV199dZLk3XffTefOnXPAAQdk5MiRmT17drbddtusvvrqGTt2bJWn5UO2W8swd+7cTJo0KZMnT87s2bMXus6uu+66gqcCqK7rr78+vXr1mu9eb7NmzcqMGTPSoUOHBda/7777ct999+WHP/zhihyzapbuIHxYhOuvvz4bbbRRRo0aNd/yefPmZa+99sopp5ySO++8M3feeWdOP/307Lbbbpk1a1aVpmVh3nnnnay77roNzx966KHMmTMnX//615Mkbdu2zR577OHm6ysZ261s8+bNy/e+972su+662WSTTfKFL3whffr0WegDoLUZMGBAbr/99vmWXXDBBQ2nDHzcI488kssvv3wFTLZycA4cy+S+++7L9OnTFzhs5IYbbsgf//jHbLDBBjnvvPOy2mqr5bLLLsvYsWPzk5/8JCeeeGJ1BmYBHTp0yNtvv93w/P77709NTU122WWXhmVt27bNe++9V43xWATbrWyDBg3KD37wg6y99to56qijst5666W21n+SVyavvvpqkn/v7W7Tpk3D86Xx4bmOAMuD/1qwTB5//PHstttuC1wk4cYbb0ylUskvfvGLhsN/9t5773Tr1i233HKLgFuJbLXVVrnzzjvzve99L23atMlNN92Uz372s/P9luuVV17JOuusU8Up+TjbrWwjRozIlltumcceeyyrrbZatcdhIbp37z7flUM/fL4klUolc+bMWQETAq2VgGOZvPHGG9l3333nWzZv3ryMHTs2G2644Xznbqy66qr5yle+krvuumtFj8linHjiiTnooIOy4YYbNuyx+d73vjffOo888kg+85nPVGlCFsZ2K9v06dNz+OGHi7eVWP/+/VOpVBou0vXhc4BqE3Ask/fee2+Bmys+//zzef/99/O1r31tgfU32GCDTJs2bUWNx1I44IAD8uMf/zhXX311KpVKDjnkkAwYMKDh9TFjxmTatGnZc889qzckC7DdytazZ89Mnjy52mOwGMOHD1/sc4BqEXAsky5duuTFF1+cb9kjjzySJNluu+0WWP+DDz5Y5AmoVM+3v/3tfPvb317oa7vttlv+9a9/reCJWBq2W7nOPPPMHHTQQXn88cftJS3ELrvskv79++eggw5Kp06dqj0O0IoJOJbJTjvtlDvvvDMvvPBCttpqq9TX12fEiBGpVCr54he/uMD6zz77bDbYYIMqTMrizJkzJ1deeWVGjhyZF154Ie+//37DORxPPvlkfvazn+Xkk0/OFltsUeVJ+SjbrVxf+cpXMnz48Oy111752te+lm222Wahl8ZO/n3oHtX3yCOP5E9/+lNOPPHEfOUrX8kRRxyRvffeO23btq32aNAiTZo0KY8++uh8z5PkscceW+Dorw9fay3cB45l8vDDD2fXXXdNhw4d0qdPn7z88st5+umn84UvfCEPPPDAfOtOnz49a6+9do488sgMGzasShPzcR988EG+9KUv5eGHH06XLl3Stm3bvPbaa5k7d26SZOrUqVl33XXzne98Z4FzrKge261sM2fOzNFHH52bbrqp4QeRj59fVV9fn0ql0rBNqa4333wzN910U2644YY8/vjjqVQqWWONNXLwwQfnsMMOy+c///lqjwgtRk1NzULPOf3w78VFLW8tf18KOJbZ8OHDc9JJJ+Xdd99Nknz2s5/NLbfcssBllH/84x/nhBNOyK233pp+/fpVYVIW5qyzzsr555+foUOH5tRTT82QIUNy3nnnzfeX4J577pm33347jz32WBUn5aNst7INHDgww4YNS8+ePXPggQcu9jYCRx555AqejiV54YUXcsMNN+Smm27KK6+8kkqlko033jhHHHFEDjvssGy22WbVHhGKdtRRRzXpfdddd10zT7JyEnA0iw8++CDPPvtsOnfunE022WSh60yYMCFTp05Njx490q5duxU8IYuyxRZbpGvXrg03Yx8yZEjOPffc+ULguOOOyy233JI33nijWmPyMbZb2dZee+1069YtY8eOdf+3wo0ZMyY33nhjbr755kybNs1tBIDlzn81aBbt27fP9ttvv9h1unfvvmKGoVFeffXV7LfffotdZ/XVV8/UqVNX0EQsDdutbDNmzEifPn3EWwuw2267pWvXrllrrbXywx/+ULwBy11NtQegXIceemhuvfXWqr2f5rH66qvnn//852LXGTduXLp06bKCJmJp2G5l++xnP5uXXnqp2mOwDN55550MGzYsO++8czbffPNcdNFFWXXVVfPNb36z2qNB0fx8uWQCjib75S9/mWeffbZq76d57LjjjrnzzjszZcqUhb4+ceLE3H333fPdlJ3qs93K9v3vfz/33ntv7rrrrmqPQiPMmjUrN998c/r165f1118/AwcOzF/+8pd89atfza9+9au8/vrr+dnPflbtMaFofr5cMsdusEyefPLJXH/99dUeg2Vw6qmnpk+fPtl9991zxRVXNBz+8/7772fs2LE54YQTMmfOnJxyyilVnpSPst3K9vvf/z69e/fOvvvum759+y7yNgKVSiVnnXVWFSbk444++ujccsstmTZtWurr67PDDjvkiCOOyCGHHJLOnTtXezyW0dy5c/OPf/wjSRa4CBsrnp8vF89FTGiympp/78Bd2OVcl+TD/9sNHjw4Z599drPOReMNGzYsJ5100kIvv9umTZtcddVVOfroo6swGYtju5Xrw78/l6Q1XRZ7ZVdTU5Pu3bvn8MMPzxFHHJHNN9+82iPRjP72t7+lR48eqampcR5jlfn5cskEHE02YsSIZf6MXr16ZZtttmmGaVhWzz//fH7yk5/kz3/+c95555106NAhn/vc53LcccflU5/6VLXHYxFstzKNGTNmqdfdbbfdluMkLK2HHnooX/jCF6o9BsvJyy+/nL59+6ZSqWT8+PHVHqdV8/Plkgk4AACAQriICQAAQCEEHAAAQCEEHMvFzJkzM3jw4MycObPao9BItl25bLsy2W7lsu3KZduVy7ZzDhzLybRp09KxY8dMnTp1oZfGZuVl25XLtiuT7VYu265ctl25bDt74AAAAIoh4AAAAApRW+0BWrN58+Zl8uTJWX311Zt0s8KV2bRp0+b7X8ph25XLtiuT7VYu265ctl25WvK2q6+vz7vvvpv111+/4YbmC+McuCqaNGlSunbtWu0xAACAlcTEiROz4YYbLvJ1e+CqaPXVV0+S7PrJk1Lbpq7K09BYZ/ziV9UegSa48OCDqj0CTTTv5VerPQJNVD9rVrVHoKn8nh9WmDmZnYdyd0MjLIqAq6IPD5usbVMn4Ar0idWdQloi/66Va16lbbVHoInqKyKgXLYdrDD/+6/bkk6t8hMoAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIQQcAABAIVpVwA0YMCCVSiUTJkxYqvVHjx6dSqWSwYMHL9e5AAAAlkaLCzjRBQAAtFS11R5gZbbDDjvk+eefz1prrVXtUQAAAATc4qy66qrZaqutqj0GAABAkhZ2COXgwYPTp0+fJMmQIUNSqVQaHh89762+vj5XXHFFttpqq9TV1aVbt24ZMmRI5s2bN9/nLepwzBdffDFHHXVUNt5449TV1WXNNdfMNttsk5NPPjn19fXL+9sEAABaqRa1B653796ZMGFCRowYkd122y29e/dueK1Tp04Nfz711FMzZsyYfPWrX82Xv/zl3H777Rk8eHBmzZqV888/f7FfY/Lkydlhhx3y3nvv5Stf+UoOPvjgvPfee3nxxRdz1VVX5eKLL05tbYv6xwoAAKwkWlRpfBhsI0aMSO/evRd5IZPHH388Tz/9dNZbb70kyVlnnZXNN988V155Zc4555y0a9dukV/jlltuyZQpU3LZZZflpJNOmu+1d955Z7HxNnPmzMycObPh+bRp05byOwMAAGhhh1AurbPOOqsh3pJkrbXWyr777pt33303f/vb35bqM9q3b7/AsjXXXHOx77ngggvSsWPHhkfXrl0bNzgAANCqtcqA++xnP7vAsg033DBJMmXKlMW+d5999sknPvGJDBw4MAcffHCuu+66vPzyy0v1dQcNGpSpU6c2PCZOnNjo2QEAgNarVQZchw4dFlj24aGPc+fOXex7u3fvnkceeST77rtv7r777nzjG9/Ipptumh49euTXv/71Yt9bV1eXDh06zPcAAABYWq0y4JbV1ltvnZtvvjnvvPNOxo4dm7PPPjuvv/56Dj744Dz88MPVHg8AAGihWlzAtWnTJsmS96Q1h7Zt22bHHXfMkCFDcsUVV6S+vj533XXXcv+6AABA69TiAu7DC4ksr/PL/vrXvy706pFvvPFGkmSVVVZZLl8XAACgRd1GIEm22mqrrL/++vnlL3+Zurq6bLjhhqlUKjnhhBOa5fNvuOGG/PSnP82uu+6aTTfdNB06dMhzzz2Xu+++O2uuuWaOOuqoZvk6AAAAH9fiAq5Nmza59dZbc/rpp2fkyJF59913kySHH354s3z+17/+9cyYMSMPP/xwHn300cycOTMbbrhhvv3tb+fUU0/NRhtt1CxfBwAA4OMq9fX19dUeorWaNm1aOnbsmL6fPi21beqqPQ6NdM5tN1Z7BJrg3H0OrfYINNG8lyZUewSaqH7WrGqPQFP5MRFWmDn1szM6v8nUqVMXe7X6FncOHAAAQEsl4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAAoh4AAAAApRW+0BSGrenJKamnbVHoNGGrL/EdUegSbo/PPJ1R6BJnrrhM2qPQJNVPvO9GqPQBPNGf9KtUcAPsYeOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIOAAAgEIIuCWYMGFCKpVKBgwYUO1RAACAVk7AAQAAFKK22gOs7DbYYIM8//zz6dixY7VHAQAAWjkBtwRt27bNVlttVe0xAAAAHEK5JAs7B+61117LSSedlM033zzt27dPp06d0qNHj3zrW9/K1KlTqzcsAADQotkD10jvv/9+dt5550yYMCFf+tKXst9++2XWrFkZP358brjhhvz3f//3Ig+3nDlzZmbOnNnwfNq0aStqbAAAoAUQcI00atSojB8/PieffHIuvfTS+V6bPn162rZtu8j3XnDBBRkyZMjyHhEAAGihHELZRO3bt19g2WqrrZa6urpFvmfQoEGZOnVqw2PixInLc0QAAKCFsQeukXbdddest956GTp0aJ566ql89atfzW677ZYePXqkUqks9r11dXWLDTwAAIDFsQeukTp27JhHHnkk/fv3zyOPPJLjjjsun/rUp9KtW7dcddVV1R4PAABowQRcE2y00UYZPnx43nzzzTzxxBO58MILM2/evAwcODAjR46s9ngAAEALJeCWQU1NTXr16pXTTjutIdzuuOOOKk8FAAC0VAKukf6//+//yxtvvLHA8g+XrbLKKit6JAAAoJVwEZNG+v3vf59TTz01O++8c7bYYot07tw5L7/8cu64446sssoqGThwYLVHBAAAWigB10hf/vKXM2HChDzwwAO59dZbM3369GywwQY5+OCDc9ppp+WTn/xktUcEAABaKAG3BN27d099fX3D8x49euSyyy6r3kAAAECr5Rw4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQgg4AACAQtRWewCSee+9l3mV2dUeg0aqfPBBtUegCSadt0W1R6CJdrv2T9UegSb6y1e6V3sEmqjStl21R6AJ6mfPqvYILEf2wAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABRCwAEAABSi1QTc6NGjU6lUMnjw4PzpT39Knz59svrqq6dLly457rjj8sEHHyRJfvvb32annXbKJz7xiayzzjo57bTTMmfOnCTJ1VdfnUqlkosuumihX+OPf/xjKpVKjj322BX2fQEAAK1Hqwm4D/35z3/O7rvvno4dO+bYY4/NRhttlGHDhuWYY47Jr371qxx44IHp1q1bjj322HTq1Ck/+MEP8v3vfz9J8vWvfz0dOnTINddcs9DP/vnPf54kOeaYY1bY9wMAALQelfr6+vpqD7EijB49On369EmS3H777dl3332TJLNnz852222XZ555Jp07d87dd9+d7bffPkny7rvvZrPNNsucOXPy+uuvp23btjnuuOMybNiwjB49OrvttlvD57/zzjtZf/3106NHjzzxxBMLnWHmzJmZOXNmw/Np06ala9eu6bv6YamttFte3zrLSaVNq/v9R4vw/k5bVHsEmmi3oX+q9gg00V++0r3aI9BEc994s9oj0AT1s2dVewSaYE797IzObzJ16tR06NBhkeu1up9A+/Tp0xBvSdK2bdsceOCBqa+vzz777NMQb0my+uqr56tf/WreeeedTJo0KUnyrW99K8m/D6f8qBtuuCEzZ85c7N63Cy64IB07dmx4dO3atTm/NQAAoIVrdQHXq1evBZatt956S3xt8uTJSZKePXtmxx13zM0335wpU6Y0rHfNNddk1VVXzWGHHbbIrz1o0KBMnTq14TFx4sSmfyMAAECr0+oCbmG7I2tra5f42uzZsxuWHXvssZkxY0ZuvPHGJP8+r+6ZZ57JQQcdlI4dOy7ya9fV1aVDhw7zPQAAAJZWqwu45nDwwQenU6dODYdRfvi/Ll4CAAAsTwKuCdq3b5/+/fvnqaeeyv33359f/epX6dGjR3beeedqjwYAALRgAq6JPrzX2+GHH553333X3jcAAGC5E3BN9MlPfjK77LJLJk+enLq6uvTv37/aIwEAAC2cgFsGRx55ZJJkv/32S+fOnas8DQAA0NLVVnuAFaV3795Z1D3LBwwYkAEDBiz0tcGDB2fw4MELfe3DG3Y7fBIAAFgR7IFrojfffDMjRozIlltumT59+lR7HAAAoBVoNXvgmstvf/vbPP7447n55pszffr0DB48OJVKpdpjAQAArYCAa6Rf//rXGTFiRNZff/18//vfzyGHHFLtkQAAgFZCwDXS8OHDM3z48GqPAQAAtELOgQMAACiEgAMAACiEgAMAACiEgAMAAChEowOub9++mTJlygLLp02blr59+zbHTAAAACxEowNu9OjRmTVr1gLLZ8yYkQcffLBZhgIAAGBBS30bgaeffrrhz88991xef/31hudz587Nvffemw022KB5pwMAAKDBUgdcr169UqlUUqlUFnqoZPv27XPllVc263AAAAD8x1IH3Pjx41NfX59NNtkkjz76aLp06dLwWrt27bL22munTZs2y2VIAAAAGhFw3bp1S5LMmzdvkevU19enUqks+1QAAAAsoNEXMRkwYEDee++9BZZPmDAhu+66a7MMBQAAwIIaHXBPPfVUevbsmbFjxzYsGzFiRLbZZpustdZazTocAAAA/7HUh1B+6NFHH83//M//pHfv3vnOd76Tl156Kffcc09++MMf5phjjlkeMwIAAJAmBFzbtm3zgx/8IKuuumrOO++81NbWZsyYMdlpp52Wx3wAAAD8r0YfQjl79ux85zvfyYUXXphBgwZlp512yv7775+77757ecwHAADA/2r0Hrjtttsu77//fkaPHp0dd9wx9fX1ueiii7L//vvnG9/4Rq666qrlMScAAECr1+g9cNttt12efPLJ7LjjjkmSSqWS008/PWPHjs0DDzzQ7AMCAADwb43eA3fNNdcsdPm2226bv/71r8s8EAAAAAvX6D1wSXLDDTdk5513zvrrr59XXnklSXLZZZfl3nvvbdbhAAAA+I9GB9ywYcNyyimnZO+9986UKVMyd+7cJEmnTp1y2WWXNfd8AAAA/K9GB9yVV16Zn//85znzzDPTpk2bhuXbbbddnnnmmWYdDgAAgP9odMCNHz8+22677QLL6+rq8t577zXLUAAAACyo0QG38cYb58knn1xg+b333psePXo0x0wAAAAsRKOvQnnKKadk4MCBmTFjRurr6/Poo49m5MiRueCCC3L11VcvjxkBAABIEwLu6KOPTvv27fPd734377//fg499NCsv/76ufzyy3PIIYcsjxkBAABIEwIuSQ477LAcdthhef/99zN9+vSsvfbazT0XAAAAH9Poc+D69u2bKVOmJElWXXXVhnibNm1a+vbt26zDAQAA8B+NDrjRo0dn1qxZCyyfMWNGHnzwwWYZCgAAgAUt9SGUTz/9dMOfn3vuubz++usNz+fOnZt77703G2ywQfNOBwAAQIOlDrhevXqlUqmkUqks9FDJ9u3b58orr2zW4QAAAPiPpQ648ePHp76+PptsskkeffTRdOnSpeG1du3aZe21106bNm2Wy5AAAAA0IuC6deuWJJk3b95yGwYAAIBFa/RFTAAAAKgOAQcAAFAIAQcAAFAIAQcAAFCIRgfcJptskrfffnuB5VOmTMkmm2zSLEMBAACwoEYH3IQJEzJ37twFls+cOTP/+Mc/mmUoAAAAFrTUtxG44447Gv583333pWPHjg3P586dm1GjRqV79+7NOhwAAAD/sdQB169fvyRJpVLJkUceOd9rbdu2Tffu3XPJJZc063AAAAD8x1IH3Ic38N54443z2GOPZa211lpuQwEAALCgpQ64D40fP355zAEAAMASNDrgzj333MW+fvbZZzd5mNZq3rvTM6/SttpjQKuwyu+fqPYINNED9Z+v9gg00ZAHfl7tEWiioTvsXu0RaIK5b79T7RFokkpSv+S1Gh1wt91223zPZ8+enfHjx6e2tjabbrqpgAMAAFhOGh1wTzyx4G+vp02blgEDBmS//fZrlqEAAABYUKPvA7cwHTp0yJAhQ3LWWWc1x8cBAACwEM0ScEkyderUTJ06tbk+DgAAgI9p9CGUV1xxxXzP6+vr89prr+WGG27IXnvt1WyDAQAAML9GB9yll1463/Oampp06dIlRx55ZAYNGtRsgwEAADA/94EDAAAoxDKdAzdx4sRMnDixuWYBAABgMRodcHPmzMlZZ52Vjh07pnv37unevXs6duyY7373u5k9e/bymBEAAIA04RDKE044Ibfeemsuuuii7LTTTkmSsWPHZvDgwXn77bczbNiwZh8SAACAJgTcTTfdlF/+8pfzXXGyZ8+e6dq1a77+9a8LOAAAgOWk0YdQ1tXVpXv37gss33jjjdOuXbvmmAkAAICFaHTAHX/88TnvvPMyc+bMhmUzZ87M+eefn+OPP75ZhwMAAOA/Gn0I5RNPPJFRo0Zlww03zDbbbJMkeeqppzJr1qzsvvvu2X///RvWvfXWW5tvUgAAgFau0QHXqVOnHHDAAfMt69q1a7MNBAAAwMI1OuCuu+665TEHAAAAS9Doc+D69u2bKVOmLLB82rRp6du3b3PMBAAAwEI0OuBGjx6dWbNmLbB8xowZefDBB5tlKAAAABa01IdQPv300w1/fu655/L66683PJ87d27uvffebLDBBs07HQAAAA2WOuB69eqVSqWSSqWy0EMl27dvnyuvvLJZhwMAAOA/ljrgxo8fn/r6+myyySZ59NFH06VLl4bX2rVrl7XXXjtt2rRZLkMCAADQiIDr1q1bkmTevHnLbRgAAAAWrdG3Ebj++usX+3r//v2bPAwAAACL1uiAO+mkk+Z7Pnv27Lz//vtp165dVl11VQEHAACwnDT6NgL/+te/5ntMnz49f/vb3/KFL3whI0eOXB4zAgAAkCYE3MJsvvnmGTp06AJ75wAAAGg+zRJwSVJbW5vJkyc318cBAADwMY0+B+6OO+6Y73l9fX1ee+21/OhHP8rOO+/cbIMBAAAwv0YHXL9+/eZ7XqlU0qVLl/Tt2zeXXHJJc80FAADAxzQ64NwHDgAAoDqafA7cW2+9lbfeeqs5ZwEAAGAxGhVwU6ZMycCBA7PWWmtlnXXWyTrrrJO11lorxx9/fKZMmbKcRgQAACBpxCGU77zzTnbaaaf84x//yGGHHZYePXokSZ577rkMHz48o0aNyp/+9KesscYay21YAACA1mypA+7cc89Nu3btMm7cuKyzzjoLvPalL30p5557bi699NJmHxIAAIBGHEJ5++235+KLL14g3pJk3XXXzUUXXZTbbrutWYcDAADgP5Y64F577bV86lOfWuTrW2+9dV5//fVmGQoAAIAFLXXArbXWWpkwYcIiXx8/fnzWXHPN5pgJAACAhVjqgPvyl7+cM888M7NmzVrgtZkzZ+ass87Knnvu2azDAQAA8B+NuojJdtttl8033zwDBw7MVlttlfr6+jz//PO56qqrMnPmzNxwww3Lc1YAAIBWbakDbsMNN8zYsWNz3HHHZdCgQamvr0+SVCqV7LHHHvnRj36Url27LrdBAQAAWrulDrgk2XjjjXPPPffkX//6V1588cUkyWabbebcNwAAgBWgUQH3oTXWWCM77LBDc88CAADAYiz1RUwAAACoLgEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQiFYTcBMmTEilUsmAAQMalg0YMCCVSiUTJkyYb93Zs2dn8ODB2XzzzVNXV5dKpZLbb799hc4LAADwcU26D1xLd8kll2TIkCHZdddd83/+z/9J27Zts9VWW1V7LAAAoJVr1QF3wQUX5IwzzsgGG2ww3/K77rorq622Wn7/+9+nXbt2VZoOAABgfq064NZbb72st956CyyfPHlyOnfuLN4AAICVSqs5B25hPn4O3ODBg1OpVDJ+/Pi88sorqVQqqVQq6d69+3zve+CBB7LPPvtkrbXWSl1dXTbffPN897vfzfvvv7/ivwkAAKDVaNV74D6ud+/eSZLLLrssSXLyyScnSTp16tSwzrBhwzJw4MB06tQp++yzT9Zee+385S9/yfnnn5/7778/999/vz13AADAciHgPqJ3797p3bt3hg8fnuTfe+Q+6rnnnsuJJ56Ynj17ZtSoUencuXPDa0OHDs2gQYNy5ZVX5jvf+c5CP3/mzJmZOXNmw/Np06Y1+/cAAAC0XK36EMrG+ulPf5o5c+bkyiuvnC/ekuS0005Lly5dMnLkyEW+/4ILLkjHjh0bHl27dl3eIwMAAC2IPXCN8MgjjyRJ7rvvvowaNWqB19u2bZsXXnhhke8fNGhQTjnllIbn06ZNE3EAAMBSE3CN8M477yRJzj///Ca9v66uLnV1dc05EgAA0Io4hLIROnTokOTfe87q6+sX+QAAAFgeBFwjfO5zn0vyn0MpAQAAViQB1wjHHXdcamtrc8IJJ+TVV19d4PUpU6bkiSeeqMJkAABAa+AcuEbYeuutc9VVV+Xb3/52ttxyy+y9997ZdNNN8+677+bll1/OmDFjMmDAgPzkJz+p9qgAAEALJOAa6ZhjjkmvXr3ywx/+MA888EDuvPPOdOzYMRtttFH+67/+K0ceeWS1RwQAAFqoVhNw3bt3X+ACI8OHD2+4afdHTZgwYbGftf322y/2fm8AAADLg3PgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAACiHgAAAAClFb7QFIKrW1qVRsitLUz5lT7RFogvp59dUegSZa9Zl/VHsEmmjoV/9PtUegiT436tlqj0ATPDRwh2qPQBPMmzMjGXv7EtezBw4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQAg4AAKAQtdUeoDWZOXNmZs6c2fB82rRpVZwGAAAojT1wK9AFF1yQjh07Njy6du1a7ZEAAICCCLgVaNCgQZk6dWrDY+LEidUeCQAAKIhDKFegurq61NXVVXsMAACgUPbAAQAAFELAAQAAFELANZNx48blhRdeyOzZs6s9CgAA0EIJuGay++67p0ePHvnHP/5R7VEAAIAWSsABAAAUwlUom8mECROqPQIAANDC2QMHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQCAEHAABQiNpqD0BSP68+9ZX6ao9BY1Uq1Z6Appg3t9oT0ERz/jG52iPQVJP9fVmqh47/XLVHoAl+//+uq/YINMG0d+dljS2WvJ49cAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIUQcAAAAIWoasANHz48lUql4XHIIYdUc5xGmzFjxnzzVyqVao8EAAC0YCvFHrh9990355xzTg488MCGZQMGDFggjj7+GD58eMP6H43BCy64YKFfZ+jQoQu8L0m6d+8+3+fW1dWlS5cu2WGHHTJw4MA89NBDC/282tranHPOOTnnnHPSrVu3Zf7nAAAAsDi11R4gSfr165cBAwYs9LVvfvOb2XDDDRf6Wq9evRa6/MILL8yxxx6bNddcc6lnaNOmTb773e8mSebMmZN//etfeeaZZ/LTn/40V111VfbZZ5+MGDEia6yxRsN7amtrM3jw4CTJ6NGj88orryz11wMAAGislSLgFufoo4/OjjvuuNTrb7rpphk3blzOP//8XHLJJUv9vo/G2Ee98sor+eY3v5k777wz++23X/74xz+mpmal2HEJAAC0Mi2uRAYMGJDNNtssP/7xj/Pqq68u8+d169Ytd955Z3r06JExY8bk5ptvboYpAQAAGq/FBVxtbW3OP//8zJw5M2eddVazfGb79u3z3//930mSX/3qV03+nJkzZ2batGnzPQAAAJbWSn8I5dVXX5177713oa+dccYZWWWVVRZYftBBB+Xiiy/OjTfemO985zvp2bPnMs/Ru3fvJMljjz3W5M+44IILMmTIkGWeBQAAaJ1W+oC75pprFvnaySefvNCAq1QqufDCC9O3b9+cccYZufvuu5d5jvXXXz9J8tZbbzX5MwYNGpRTTjml4fm0adPStWvXZZ4NAABoHVb6QyjHjh2b+vr6hT46deq0yPf16dMne+65Z+65556MGTNmxQ28GHV1denQocN8DwAAgKW10gfcshg6dGhqampy2mmnLfNnTZ48OUnSpUuXZf4sAACApmjRAbfNNtvksMMOy6OPPppf//rXy/RZo0ePTpJsv/32zTAZAABA47XogEuS8847L3V1dTnzzDMzZ86cJn3GBx980HBPua9//evNOR4AAMBSa/EB161btxx33HF58cUXM3z48Ea//9VXX80+++yT5557Ln369Mn+++/f/EMCAAAshZX+KpSLu43AjjvumD333HOJn3HmmWfm2muvzbhx4xa5zpw5czJ48OAkydy5czNlypQ8/fTTefjhhzN37tzsu+++GT58eCqVSpO+DwAAgGW10gfc4m4jcNJJJy1VwHXu3Dmnn356/ud//meR68ydO7fhHm3t2rVLhw4dsvHGG+fYY4/NoYcemp133rnxwwMAADSjlTbghg8f3qhDHgcMGJABAwYs8vVBgwZl0KBBC31twoQJjRsOAACgClaKc+COOuqoVCqVHHLIIdUepVFmzJiRSqWSSqWy0txrDgAAaLmqugeuV69eOeeccxqeb7311lWcpvFqa2vnmx8AAGB5qnrA9erVq5ojLJPa2tqGC58AAAAsbyvFIZQAAAAsmYADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAohIADAAAoRG21B2jN6uvrkyRz6mdXeRKapH5etSegKf733ztgRapUewCaaN6cGdUegSaY9q6fUUo0bfq/t1v9En5WqdQvaQ2Wm0mTJqVr167VHgMAAFhJTJw4MRtuuOEiXxdwVTRv3rxMnjw5q6++eiqVlvXbyWnTpqVr166ZOHFiOnToUO1xaATbrly2XZlst3LZduWy7crVkrddfX193n333ay//vqpqVn0mW4OoayimpqaxdZ1S9ChQ4cW9y9Xa2Hblcu2K5PtVi7brly2Xbla6rbr2LHjEtdxERMAAIBCCDgAAIBCCDiWi7q6upxzzjmpq6ur9ig0km1XLtuuTLZbuWy7ctl25bLtXMQEAACgGPbAAQAAFELAAQAAFELAAQAAFELAAQAAFELAAQAAFELAAdDqDRgwIP369VvhX3f48OHp1KnTEtcbPHhwevXq1exff3l9LgDLj4ADAAAohIADgI/p3bt3TjzxxJx22mlZc801s+6662bw4MHzrVOpVDJs2LDstddead++fTbZZJPcfPPNDa+PHj06lUolU6ZMaVj25JNPplKpZMKECRk9enSOOuqoTJ06NZVKJZVKZYGvkfx7L92QIUPy1FNPNaw3fPjwJMmUKVNy9NFHp0uXLunQoUP69u2bp556Kkny5ptvZt111833v//9hs/605/+lHbt2mXUqFGL/VwAVl611R4AAFZGI0aMyCmnnJI///nPGTt2bAYMGJCdd945e+yxR8M6Z511VoYOHZrLL788N9xwQw455JA888wz6dGjxxI///Of/3wuu+yynH322fnb3/6WJFlttdUWWO/ggw/Os88+m3vvvTd/+MMfkiQdO3ZMkhx00EFp37597rnnnnTs2DE//elPs/vuu+fvf/97unTpkmuvvTb9+vXLl770pWy55ZY54ogjcvzxx2f33XfPBx98sMjPBWDlJeAAYCF69uyZc845J0my+eab50c/+lFGjRo1X8AddNBBOfroo5Mk5513Xn7/+9/nyiuvzFVXXbXEz2/Xrl06duyYSqWSddddd5HrtW/fPquttlpqa2vnW++hhx7Ko48+mn/+85+pq6tLklx88cW5/fbbc/PNN+f//t//m7333jvHHHNMDjvssGy33Xb5xCc+kQsuuGCxnwvAyk3AAcBC9OzZc77n6623Xv75z3/Ot2ynnXZa4PmTTz65vEdLkjz11FOZPn16OnfuPN/yDz74IOPGjWt4fvHFF2frrbfOr3/96/z1r39tiD0AyiTgAGAh2rZtO9/zSqWSefPmLfX7a2r+fZp5fX19w7LZs2c3z3BJpk+fnvXWWy+jR49e4LWPXtly3LhxmTx5cubNm5cJEybk05/+dLPNAMCKJ+AAoIkeeeSR9O/ff77n2267bZKkS5cuSZLXXnsta6yxRpIssHeuXbt2mTt37hK/zsLW+8xnPpPXX389tbW16d69+0LfN2vWrBx++OE5+OCDs+WWW+boo4/OM888k7XXXrtRXx+AlYerUAJAE/3617/Otddem7///e8555xz8uijj+b4449Pkmy22Wbp2rVrBg8enBdffDG//e1vc8kll8z3/u7du2f69OkZNWpU3nrrrbz//vsL/Trdu3fP+PHj8+STT+att97KzJkz88UvfjE77bRT+vXrl9/97neZMGFC/vSnP+XMM8/MX/7ylyTJmWeemalTp+aKK67I6aefni222CLf+MY3Fvu5AKzcBBwANNGQIUPyy1/+Mj179sz111+fkSNH5pOf/GSSfx+COXLkyLzwwgvp2bNnLrzwwnzve9+b7/2f//zn861vfSsHH3xwunTpkosuumihX+eAAw7InnvumT59+qRLly4ZOXJkKpVK7r777uy666456qijssUWW+SQQw7JK6+8knXWWSejR4/OZZddlhtuuCEdOnRITU1Nbrjhhjz44IMZNmzYIj8XgJVbpf6jB+cDAEulUqnktttuS79+/ao9CgCtiD1wAAAAhRBwAAAAhXAVSgBoAmcgAFAN9sABAAAUQsABAAAUQsABAAAUQsABAAAUQsABAAAUQsABAAAUQsABAAAUQsABAAAU4v8HSSKG3V1OaakAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%time\n",
"# This is my life.\n",
"model.plot_attention('Esta es mi vida.')"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:11.214924Z",
"iopub.status.busy": "2023-12-07T12:24:11.214656Z",
"iopub.status.idle": "2023-12-07T12:24:11.618813Z",
"shell.execute_reply": "2023-12-07T12:24:11.618101Z"
},
"id": "t-fPYP_9K8xa"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmpfs/tmp/ipykernel_16852/3355722706.py:23: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_xticklabels([''] + context, fontdict=fontdict, rotation=90)\n",
"/tmpfs/tmp/ipykernel_16852/3355722706.py:24: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_yticklabels([''] + output, fontdict=fontdict)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 220 ms, sys: 63.1 ms, total: 283 ms\n",
"Wall time: 196 ms\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2wAAAOTCAYAAADDqICdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOiklEQVR4nO3deZzWdb3//+c1g+DKgAoogiBSqUcFS1IzFVGL9FTkUqapo+nPTi6ZnjRSZDFDLU+apqdyAbWw3CrLpULBJU3LLVNLEQj3tGDcGGCY3x9+nSMBCjoz15uZ+/12m5tzfT6f6zMvuMZxHn6Wq9Lc3NwcAAAAilNT7QEAAABYNsEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGAABQKMEGALS5CRMm5PLLL6/2GACrHMEGALS5b37zm/nzn/9c7TEAVjmCDQBocxtvvHHmzp1b7TEAVjmCDQBoc/vvv39uuummzJs3r9qjAKxSKs3Nzc3VHgIA6NgaGxuzzz775Nlnn82ECRMybNiw9O7du9pjARRPsAEAba62tjZJ0tzcnEqlstztKpVKFi1a1F5jARSvS7UHAAA6vp122ultQw2AZXOEDQAAoFBuOgIAAFAowQYAAFAo17ABAK3usMMOS6VSybe+9a306dMnhx122Ao9r1Kp5OKLL27j6QBWHa5hAwBaXU1NTSqVSh599NG8//3vT03Nip3UU6lU0tTU1MbTAaw6HGEDAFrdzJkzkyQbbbTREo8BWDmOsAEAABTKTUcAgDY3aNCgHHXUUdUeA2CVI9gAgDb34osvpnv37tUeA2CVI9gAgDa39dZb529/+1u1xwBY5Qg2AKDNnXTSSbn++utz6623VnsUgFWKu0TSpkaMGPGe91FfX5+DDz64FaYBoFr+9a9/5WMf+1g+9rGPZdSoURk2bFj69OmTSqWy1LZ+5gP8H3eJpE2t6PvuLE+lUsnYsWNz6qmnttJEAFTDm+/L9u+/drw12Jqbm70PG8C/cYSNNjdu3Lh3HVzvNfgAKMOll15a7REAVkmCDQBoc4cccki1RwBYJQk22tSNN96YwYMHV+35AACwKnMNGwDQbu67775Mnjw5999/f+bNm5e6urp88IMfzMEHH5wPfvCD1R4PoDiCjTY3aNCgHHfccTn22GOrPQoAVfS1r30t3/3ud7N48eKl1tXU1OT444/PWWedVYXJAMrljg60uVmzZmXu3LnVHgOAKjr//PNz9tln533ve18uv/zyzJo1K6+//npmzZqVyy67LIMHD87ZZ5+dCy64oNqjAhTFETbaXE1NzXu6UyQAq74tttgir776ah5++OGss846S62fN29ettpqq6y99tp55JFHqjAhQJkcYQMA2tzMmTOzzz77LDPWkqSuri777LNPZs6c2c6TAZRNsNEu3vrGqAB0Pr17916h7fr06dPGkwCsWpwSSZurqalJjx490qNHjxV+TqVSyYwZM9puKADa1de//vVMmTIlf/nLX7L22msvtb6hoSFbbrllDjzwwEycOLEKEwKUSbDR5mpq3t2B3GXdRQyAVVNjY2M++9nP5vHHH8+pp56aj370o+nTp0+ef/753H777TnttNPy/ve/Pz/72c/StWvXao8LUAzBRptz0xGAzqempmaZp8M3Nze/7fJKpZJFixa1x4gAq4Qu1R4AAOh4dt55Z9cvA7QCwQYAtLpp06ZVewSADsFdIgEAAAol2CjO4sWLM2nSpGqPQSv7+9//nueee67aYwAArFLcdIQ2N3v27PTo0SN1dXXvuO2Pf/zjTJgwIU888USampraYTraS21tbQ455JBccskl1R4FqIIRI0as0HaVSiVTp05t42kAVh2uYaPNDRgwIC+99FLGjx+fP/3pT+nSpUt22mmnfOlLX8oaa6yRJPnlL3+Z0aNH57HHHkuS7L333tUcmTbQs2fPrLfeetUeA6iSd7qmrVKpLPcOksCqqba29j3vY+zYsZ3+TuOCjTb3/PPP58Mf/nCeeuqpvHlA9xe/+EWuu+66TJ06NYcffniuuOKKJMmoUaMyduzYbL311tUcmTaw00475Q9/+EO1xwCqZHnvrdnQ0JD77rsv3/jGN9KvX79MmTKlnScD2kpzc3MGDBiQgQMHvqvn3nbbba0/1CpIsNHmTj/99MyZMyd77rln6uvrkySXXHJJbr755uy222654447svPOO+fcc8/NkCFDqjssbWbixInZfvvtM2HChHzjG99Ily5+/ABJ9+7dM3z48Nx8883Zaqutcvrpp3f6/5sOHcmhhx76rv+drqlxu41EsNEObr755myxxRb51a9+1bJsn332yZZbbpk777wzBx98sJuMdAJnnXVWttpqq4wfPz4/+MEPMmTIkPTp02ep058qlUouvvjiKk0JVMs666yTT3ziE7n00ksFG8BbCDba3Jw5c3LEEUcssaxSqWT33XfPY489lnHjxlVnMNrVW6P82WefzbPPPrvM7QQbdF41NTXL/dkArHpef/3193RGzXt9fkfhb4A2N3/+/GXebGLddddNknd1XjOrnpkzZ1Z7BKBgTz75ZK666ir/TYAOpFu3blV9fkch2IB2MWDAgGqPAFTRYYcdtszlixYtytNPP5077rgjCxcuzIQJE9p5MoCyeR822lxNTU1233337L777kss/+1vf5tbbrklZ5xxRpb1bXjiiSe214gAtLF3unnABz7wgZxwwgk5/PDD22kioL1dffXV+elPf5p77703L774YpKkV69eGTZsWA444ICMGjWqugMWSrDR5t7NHX4qlYo3zu6g5s+fn3vvvTfPPPNMGhsbl7nNwQcf3M5TAW1t9uzZy1xeU1OTHj16ZJ111mnniYD2Mnfu3Oy9996ZPn36Mv8nffLG734jRozINddck+7du7fzhGUTbLS5yZMnv6vnHXLIIa08CdX2/e9/P2PGjMm8efOWuf7NN80V6wDQcXz605/O9ddfn8033zz//d//neHDh6dv375JkmeeeSa33nprzj777Dz66KP5zGc+k2uuuabKE5dFsAHt4tprr82+++6brbbaKvX19TnhhBMyatSobLfddrntttty4403Zp999sl//ud/inXogJqamvLqq69m7bXXXuaZF2+uX2uttVJbW1uFCYG2cOedd2annXbK7rvvnl/+8pdZffXVl7nd/Pnz85//+Z+59dZbc8cdd2SHHXZo50nL5d3oKM6CBQty5ZVXVnsMWtk555yT3r1756677spXv/rVJMnQoUNz0kkn5de//nWuuOKK/PznP3dzEuigxo8fn969e+ell15a5vp//vOf6dOnT04//fR2ngxoS1OmTEmXLl1y8cUXLzfWkmT11VfPJZdckpqamkyZMqUdJyyfYKMYDz74YI499thsuOGGOfDAA6s9Dq3soYceyqc+9amsueaaLcveeurjAQcckBEjRrhDHHRQv/rVr7LbbrulV69ey1zfq1ev7L777vnFL37RzpMBbenee+/NRz7ykfTv3/8dt914442z44475p577mmHyVYdgo2qmjdvXi688MJsu+22+eAHP5jzzz8/8+fPzwEHHFDt0WhlCxcuXOIXtTXWWCNz585dYpshQ4bkvvvua+fJgPbw5JNPZrPNNnvbbT7wgQ94z0boYGbOnJmtttpqhbffeuut/Rz4N96Hjaq49dZbc/HFF+e6667L/Pnzk7zxBtpf//rX87nPfc7dgTqgvn375tlnn215PGDAgNx///1LbDN79ux06eLHEnRECxcufMe7BlcqlZb/JgAdQ0NDQ3r27LnC2/fo0SMNDQ1tONGqxxE22s3TTz+d008/PYMHD87uu++en/zkJ1l//fVz0kknpbm5ObvttluOOOIIsdZBDRs2bImjZyNHjsydd96ZiRMn5i9/+Ut+8IMf5Nprr82wYcOqOCXQVgYPHpxbbrnlbbe55ZZbsskmm7TTREB7WLBgwUr9z9ja2tosWLCgDSda9Qg22tw111yTvfbaKwMHDsyYMWPywgsv5KCDDsrvfve7zJo1K9/61reqPSLtYL/99ktjY2NmzZqVJBk9enT69euXU045JVtvvXX+67/+K2uvvXbOOuus6g4KtIm99947DzzwQE499dSl3rqjqakpY8aMyQMPPJD99tuvShMCbaVSqVR7hFWa2/rT5mpqalJTU5Pdd989Bx10UPbee++sscYaS21z+OGH54c//GGVpqQa/vWvf+Wiiy7Kk08+mQEDBuSggw7KRhttVO2xgDbwyiuvZNiwYfnb3/6WTTfdNLvuums22mijPP3007n11lszY8aMbL755rn77ruz9tprV3tcoJXU1NSkS5cuK3yUbdGiRWlqavKerG/hYhHaRXNzc55//vn84x//yMsvv7xUsNE59ezZM1/72teqPQbQDtZee+3cdttt+a//+q9cd911eeKJJ1rW1dTUZN99980FF1wg1qCD2XjjjR1he48cYaPN3X777bnoootyzTXX5LXXXkuXLl2yxx575OCDD86oUaPSrVs3R9g6gREjRqS+vj4HH3zwcre54oorcskll7zjdS7Aqu3555/PH//4x8ybNy89evTItttum969e1d7LIAiOcJGm9tpp52y00475fzzz89PfvKTXHzxxbnxxhtz0003ZZ111sm+++5b7RFpB9OmTcvw4cPfdpvZs2dn+vTp7TMQUDV9+vTJXnvtVe0xAFYJbjpCu1lnnXVy5JFH5p577slDDz2UY445JquttlouueSSJG+8qeqECRMye/bsKk9Ktbz66qtZbbXVqj0G0EYWL1681LK77rorJ598ck477bQ89dRTVZgKoGxOiaSqFixYkOuuuy6XXHJJpk6dmubm5lQqleyyyy6ZOnVqtcfjPfr73//e8vnAgQNz3HHH5bjjjltqu6ampsyZMydHHHFEKpVKHnvssXacEmgPX/3qV3PhhRfmueeeS48ePZIkV199dfbff/+WkFt//fVz3333pV+/flWcFGhNl1122bt63ttdQtHZCDba3KBBg3Lcccfl2GOPfdvt/v73v+eSSy7JpZdemqeeesrdgTqAmpqalbrQuLm5Od/+9rdzwgkntOFUQDUMHTo0ffv2zQ033NCybIsttsjzzz+fc889N88991xGjx6do446Kuecc071BgVa1Yr+LvDm/7RP3ngbgEWLFrX1aKsM17DR5mbNmpW5c+e+43Ybb7xxxo0bl7Fjx+Z3v/td2w9Gmzv44INTqVTS3Nycyy67LEOGDMnQoUOX2q62tjbrrrtuRowYkZEjR7b/oECbmzNnTnbZZZeWxzNnzsxjjz2WsWPH5gtf+EKSN25SddNNN1VrRKANfOtb31qhYFu0aFEmTZqUGTNmtMNUqxbBRnEqlUr22GOPao9BK5g0aVLL59OnT8+hhx76jkdagY7p1VdfzVprrdXyePr06alUKvnEJz7RsmyLLbZwOjx0MF//+tffcZurr746p5xySmbMmJG6urqcdNJJ7TDZqkOwAe1i5syZ1R4BqKK+ffvmr3/9a8vjm266KWuvvXY+9KEPtSxraGhIt27dqjEeUAW33357TjzxxNxzzz3p2rVrvvrVr+bkk0/OuuuuW+3RiiLYaBfeMBGgc9tll10yZcqUnH/++Vl99dVz7bXXZtSoUamtrW3ZZsaMGW44Ap3Aww8/nK9//eu58cYbU6lU8oUvfCGnnXZaNt5442qPViQ3HaHN1dTUpEePHi13BVsRlUrFOcwd0Msvv5zzzz8/v/vd7/LMM8+ksbFxqW289tAxPfHEExk2bFgaGhrS3NyctdZaK3/4wx+yxRZbJHnj50OfPn1SX1+fCy64oMrTAm3hqaeeypgxY3LFFVekqakpI0eOzBlnnJGtt9662qMVzRE22sXcuXNX6MYjdFz/+Mc/8pGPfCQzZsxI9+7d09DQkLq6uixYsCCvv/56kjdOmfI+bNAxDR48OI888kiuueaaJMknP/nJDBgwoGX9448/niOPPDIHHHBAtUYE2sjcuXNz+umn5/vf/37mz5+fYcOG5cwzz8zw4cOrPdoqwRE22lxNTU3GjRuXU089tdqjUEVHHXVULrzwwlx22WU58MADU1tb2/J9ce+99+aYY45Jly5d8pvf/CZrrrlmtccFAFrBmWeemTPPPDPz5s3Lpptumm9961vZd999qz3WKsURNqBd3HDDDdltt91abt/9VsOGDcuNN96YrbbaKuPHj8+ZZ55ZhQlpT4888kgee+yxvPrqqznooIOqPQ7tzOsPncfo0aNTqVSy6aabpr6+Pk8++WTOOuusd3zeiSee2A7TrRoEG9Aunn322ey3334tj2tra1tOhUySnj175hOf+ER+9rOfCbYO7N57780RRxyRP//5zy3L3vyF/bbbbsvIkSNz5ZVX5lOf+lS1RqQNef2hc2pubs4TTzyRMWPGZEVO7qtUKoLtLQQb0C7q6uqycOHClsc9e/bMU089tcQ23bt3z/PPP9/eo9FO/vKXv2TEiBGpqanJV7/61Tz22GO58cYbW9bvtNNOWX/99XPVVVf5hb0D8vpD53TppZdWe4RVnmAD2sWgQYMya9aslsfbbLNNfvvb3+all17Keuutl9dffz3XX3+9W/p2YGPHjk2S/OlPf8rgwYMzfvz4JX5hr1Qq2WGHHXLvvfdWa0TakNcfOqdDDjmk2iOs8mqqPQAd38yZM/OVr3yl2mNQZR/72McyderUvPbaa0mSI488Mi+88EKGDBmS/fbbL1tuuWVmzJiR+vr66g5Km5k+fXr22WefDB48eLnbbLzxxnn22WfbcSrai9cf4N0RbLS5AQMGpK6ubpnrFi5cmHPPPTef/vSn8+lPfzrf+c53Mn/+/HaekPbwpS99KT/60Y9agm3vvffOt7/97bz66qu55ppr8txzz+X444/P1772tSpPSlt5+eWX07t377fd5vXXX09TU1M7TUR78vpD53TZZZfloYceWmLZggUL0tDQsMztb7755hx//PHtMdoqQ7DR5i677LJsvPHGmTp16hLLFy9enE984hM5/vjjc/311+f666/PSSedlF122SULFiyo0rS0lQ033DCf+9znsv7667csO+GEE/Liiy/m2WefzSuvvJJvf/vbqa2treKUtKX+/fsvcbOJZbnvvvuy6aabttNEtCevP3RO9fX1+fnPf77EsokTJ6Znz57L3P7uu+/Oueee2w6TrToEG23u5ptvziuvvLLUmyNefvnlueWWW9K3b99ccskl+dnPfpYddtghf/zjH/O///u/1RmWNnPYYYflu9/97lLLa2tr06dPn1QqlSpMRXv6z//8z/zmN7/J7373u2Wu/9nPfpa77747o0aNat/BaBdef4B3R7DR5u67777ssssuSx05ueKKK1KpVPLjH/849fX12XffffOb3/wm6667bq655poqTUtb+clPfpIXXnih2mNQRd/4xjfSt2/f7LnnnjniiCPyxz/+MUlywQUX5KCDDsoBBxyQgQMHOhWmg/L6A7w77hJJm3v++efz6U9/eollixcvzl133ZV+/fpl5513blm+5pprZq+99sqvfvWr9h6TNrbpppu6mUAn16tXr0yfPj0HHXRQLr744pblRx99dJJku+22y5QpU5Z7zSurNq8/wLsj2Ghzr7766lJvkvjoo4/mtddeW+Z77Wy00UbLvRCVVddhhx2WM844I08//XQ22mijao9DlQwaNCh33nlnHnjggdx999355z//me7du2e77bbLsGHDqj0ebczrD7DyBBttrlevXnn88ceXWHb33XcnSbbddtultn/99deXeyEqq6599tknt956az7ykY/kxBNPzLBhw5Z77Zr3Yuv4hg4dmqFDh1Z7DKrE6w+w4gQbbW6HHXbI9ddfn8ceeyybbbZZmpubM3ny5FQqley+++5Lbf/www87AtMBDRo0KJVKJc3NzTn22GOXu12lUsmiRYvacTLaymGHHfaunlepVJY4ZY5Vk9cfeNNTTz2Ve+65Z4nHSXLvvfcudRbWm+v4P5Xmf/9bglZ25513Zuedd0737t2z66675sknn8xDDz2Uj370o7ntttuW2PaVV15J7969c8ghh+TCCy+s0sS0hfr6+hW+E+Sll17axtPQHmpqln1fqzfDfXnLK5WK9+LqALz+QPLGz4Jl/ff/zX/fl7fcz4H/4wgbbW7HHXfMxRdfnK985Sst78PxoQ99KFdcccVS206ePDnz58/Pxz/+8XaekrY2YcKE9OjRI927d1/uNg0NDZk7d277DUWbmjlz5hKPFy9enK985Su5++6785WvfCU77bRT+vTpk+effz633XZbvve972WHHXZY5ts/sOrx+gNJcsghh1R7hFWeI2y0m9dffz0PP/xw1ltvvQwaNGiZ28yaNSvz5s3L5ptvnq5du7bzhLSl2trajBs3LmPGjFnuNqeffnrGjh3rlMgO6owzzsh3v/vdPPDAA9lwww2XWv/0009nm222yX//93/nxBNPrMKEtCWvP8C7433YaDdrrLFGhg0bttxYS5KBAwdmyJAhYq0Dam5uXuZpUCu7Dauuiy++OJ/97GeX+ct68sYdYj/72c/mRz/6UTtPRnvw+gO8O4KNNnXAAQfk2muvrdrzWbU89dRTWWeddao9Bm3kqaeeyuqrr/6226y++uouOO+gvP7Q+fg9sHW4ho02deWVV2azzTbL3nvvXZXnU10TJkxY4vG0adOWuV1TU1PmzJmTK6+8Mttvv307TEY19OvXL9ddd11OO+20Zf7i/tprr+W6665Lv379qjAdbc3rD52P3wNbh2CjzT3wwAO57LLLqj0GVTBu3LiWzyuVSqZNm7bcaEuSvn375swzz2z7waiKww8/PKNHj86OO+6YU089NR/96Eez3nrr5aWXXsrtt9+eCRMmZNasWZk4cWK1R6UNeP15J01NTXn66aeTeD/OjsTvge+dm47Qpt68rfOK3s79rd781hw3blxOPfXUVp2L9jF9+vQkb7yWI0aMSH19/TLvFlVbW5t11103m2222XJvBc6qb/HixTniiCNy6aWXtvxMqKmpyeLFi5O88X1y6KGH5qKLLnpXPzMom9efd/LXv/41m2++eWpqatx8qoPwe2DrEGy0qcmTJ7/nfQwdOjRDhgxphWmopvHjx2fXXXfNzjvvXO1RqLLp06dn8uTJeeihhzJv3rzU1dVlyJAhOeiggzJ8+PBqj0cb8/qzPE8++WRGjBiRSqWy1NtCsGrye2DrEGwAAACFcu4RAABAoQQbAABAoQQbVdPY2Jhx48alsbGx2qNQJb4HOjevf+fm9cf3QOfm9V9xrmGjahoaGlJXV5d58+ale/fu1R6HKvA90Ll5/Ts3rz++Bzo3r/+Kc4QNAACgUIINAACgUF2qPUBntnjx4jzzzDNZZ511OuWbhDY0NCzxTzof3wOdm9e/c/P643ugc+vsr39zc3Nefvnl9O3bt+UNxpfHNWxV9NRTT6V///7VHgMAAKiCOXPmpF+/fm+7jSNsVbTOOuskSXYZ9OV0qe1W5Wmohvqf/a7aI1BFkw74eLVHoMoWPz6z2iNQRc1NTdUeAaiSRc0Lc0d+3dIDb0ewVdGbp0F2qe0m2DqpNdeprfYIVJF/71lcWa3aI1BFzRW3EoBOrTkrdFmUnxQAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACFEmwAAACF6pTBNm3atFQqlYwbN67aowAAACxXpww2AACAVYFgAwAAKFSnC7Zx48Zl1113TZKMHz8+lUql5WPWrFmpr69PpVLJk08+mbPPPjtbbLFFunXrlvr6+pxyyimpVCr52c9+tsx9X3LJJalUKpk4cWJ7/pEAAIAOqku1B2hvw4cPz6xZszJ58uTssssuGT58eMu6Hj16tHx+zDHH5O67785ee+2VT37yk+ndu3f23XffTJw4MRdddFE++9nPLrXvH/3oR+nSpUsOPfTQdviTAAAAHV2nDLYkmTx5coYPH77cG4889NBDuf/++7PxxhsvsfzjH/94brrppsyaNSsDBw5sWf6Xv/wld999d0aNGpUNNthgmftsbGxMY2Njy+OGhob39GcBAAA6tk53SuSK+trXvrZUrCXJl770pTQ3N+fiiy9eYvlFF12UJDniiCOWu8+JEyemrq6u5aN///6tOzQAANChCLbl+PCHP7zM5XvttVc22mijXHrppWlqakqSLFiwIJdffnn69++fkSNHLnefo0ePzrx581o+5syZ0yazAwAAHYNgW44+ffosc3ltbW0OP/zwPP3007nxxhuTJNddd11eeumlHHbYYampWf5fabdu3dK9e/clPgAAAJZHsC1HpVJZ7rrDDz88tbW1+dGPfpTkjdMha2pqcthhh7XXeAAAQCfQKYOttrY2SVpOaVxZ/fr1y1577ZUbbrghv//97zN16tR8/OMfX+Y1bwAAAO9Wpwy2ddddN0ne0zVkRx55ZBYtWpT99tsvzc3Nb3uzEQAAgHej093WP0k222yz9O3bN1deeWW6deuWfv36pVKp5JhjjlnhfYwcOTIDBgzI7Nmzs8EGG+STn/xkG04MAAB0Rp3yCFttbW2uvfbabL/99pkyZUpOPfXUjBkzJv/6179WeB81NTU56KCDkiT19fXp0qVTti8AANCGOmWwJcl2222XadOmpaGhIc3NzWlubs7AgQMzadKkls/fyf33359KpZIvfvGLbT8wAADQ6XTaYHuvHnnkkdxwww3ZY489Mnjw4GqPAwAAdEDO41tJP/nJT/LXv/41l112WZJk7NixVZ4IAADoqATbSvrhD3+Y22+/PQMGDMjFF1+cj3zkI9UeCQAA6KAE20qaNm1atUcAAAA6CdewAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFEqwAQAAFKpLtQcgqbw+P5Wa5mqPQRX8aI8R1R6BKnpi4urVHoEq2+CnH6z2CFTROg+9UO0RqKKmOc9UewSqqNKcZOGKbesIGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEE27+ZNm1aKpVKxo0bV+1RAACATk6wAQAAFEqwAQAAFEqwvcW4ceOy6667JknGjx+fSqXS8jFr1qwkyYsvvpjjjjsum2yySbp165bevXvns5/9bB5++OEqTg4AAHREXao9QEmGDx+eWbNmZfLkydlll10yfPjwlnU9evTIP/7xj+ywww6ZMWNGhg8fnv333z8zZ87M1VdfnV//+te5+eab89GPfrR6fwAAAKBDEWxv8WagTZ48OcOHD1/qxiOHHXZYZsyYkdGjR+db3/pWy/Ibbrghe+21Vw499ND89a9/TU3Nsg9cNjY2prGxseVxQ0NDq/8ZAACAjsMpkStowYIFmTJlStZbb72ccsopS6zbc889s8cee+SJJ57InXfeudx9TJw4MXV1dS0f/fv3b+uxAQCAVZhgW0GPPfZY5s+fnw9/+MNZc801l1r/5rVvDzzwwHL3MXr06MybN6/lY86cOW01LgAA0AE4JXIFvXn6Yp8+fZa5fsMNN1xiu2Xp1q1bunXr1vrDAQAAHZIjbCuoe/fuSZLnn39+meufe+65JbYDAAB4rwTbv6mtrU2SNDU1LbF8s802y+qrr5577703r7322lLPmzZtWpJk6NChbT0iAADQSQi2f7PuuusmyVLXl3Xt2jWf//zn8+KLL2bixIlLrLvpppty8803Z/Dgwdlxxx3bbVYAAKBjcw3bv9lss83St2/fXHnllenWrVv69euXSqWSY445JmeeeWamT5+eb37zm/n973+f7bbbLrNmzcpVV12VNddcM5deeulyb+kPAACwsgTbv6mtrc21116bk046KVOmTMnLL7+cJPnCF76QgQMH5g9/+ENOO+20/OIXv8jtt9+eurq6jBo1KmPHjs2WW25Z5ekBAICOpNLc3Nxc7SE6q4aGhtTV1WX3fv+VLjXuHtkp1Toi25k9PrFntUegyjb4qZ/9ndk6D71Q7RGooqY5z1R7BKpoUfPC3LrwqsybN+8db1rot0UAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCdan2ACSLnn42qaxW7TGogtq67tUegSoacMF61R6BKvvYhVOrPQJVdOtOG1d7BKqosppfwzuzSvPiZOGKbesIGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKEEGwAAQKGqHmwLFy7MuHHj8r73vS/dunVLpVLJOeeck0qlkvr6+mqPBwAAUDVdqj3A2WefnfHjx2fnnXfOZz/72ay22moZOXJkvvrVr1Z7NAAAgKqqerD96le/ytprr53f/va36dq1a5I3jro9+uijqaurq/J0AAAA1VP1YHvmmWey3nrrtcRakqy22mrZbLPNqjgVAABA9VXtGrZx48alUqlk5syZmT17diqVSiqVSgYOHJhZs2Yt9xq2hx56KHvuuWfWWWed1NXVZc8998zDDz+c+vr6VCqVzJo1a6mvMW3atKX2M2nSpFQqlUyaNKll2Vu/7qOPPprPfOYzWW+99Zba7y9+8Yvstttu6dmzZ1ZfffVsueWW+c53vpOmpqbW+wsCAAA6vaodYRs+fHiS5JxzzkmSHHfccUmSHj16LPc5Dz74YHbaaae8+uqr2XvvvfO+970vf/zjH/PRj340Q4YMabXZnnjiiWy//fbZaqutUl9fn5deeqnlCODo0aNzxhlnZKONNsree++durq63H777fna176WP/zhD7nqqquWu9/GxsY0Nja2PG5oaGi1mQEAgI6nqsE2fPjwliNc48aNa1n31qNZb3X00Ufn5Zdfzo9//OMccMABLctPPfXUnHbaaa0225133plTTz0148ePX2L5b3/725xxxhn5+Mc/nmuuuSZrrbVWkqS5uTlf/vKX87//+7+55pprss8++yxzvxMnTlxqnwAAAMtT9dv6r6jZs2fnjjvuyJAhQ5aItSQ56aST0rNnz1b7WhtssEFOPvnkpZaff/75SZIf/vCHLbGWJJVKJWeccUYqlUqmTJmy3P2OHj068+bNa/mYM2dOq80MAAB0PFW/6ciKevDBB5MkO+6441Lr1lprrQwdOjS33nprq3ytIUOGLHETlDfdfffdWWuttXLJJZcs83lrrLFGHnvsseXut1u3bunWrVurzAgAAHR8q0ywvXm9V+/evZe5vk+fPq32tZa3r3/+859ZtGjR257W+Oqrr7baHAAAQOe2ygRb9+7dkyQvvPDCMtc///zzSy2rqXnjjM9FixYttW7evHnL/VqVSmW5M1Qqlbz44ovvOC8AAMB7tcpcw/bmXSB///vfL7Xutddeazll8q3evK7t6aefXmrd/fffv9IzbLfddnnppZfy+OOPr/RzAQAAVtYqE2wDBgzIjjvumAceeCA//elPl1j37W9/O//85z+Xes6wYcOSJJdddlkWL17csvyuu+7Kj3/845We4dhjj02SHHbYYXnppZeWWv/cc8/l0UcfXen9AgAALMsqc0pkkpx33nnZeeedc+CBB+aaa67J4MGDc9999+Xuu+/OzjvvnNtuu63lNMgk2X777bPjjjvmlltuyQ477JCdd945s2fPzi9+8Yt88pOfzHXXXbdSX3/kyJEZM2ZMTjvttAwePDgjR47MgAED8tJLL+WJJ57I7bffnm9+85vZfPPNW/uPDgAAdEKrzBG2JNlmm21y++23Z/fdd8+NN96Y888/PzU1NbnjjjtarnF7859v+sUvfpGDDz44TzzxRL7//e9nzpw5uf766/OpT33qXc0wYcKE/Pa3v81OO+2UqVOn5n/+53/yq1/9Ko2NjRk3blwOPPDA9/znBAAASJJKc3Nzc7WHeK+ampqy6aab5vXXX1/mzUdK1dDQkLq6ugyvjEqXymrVHocqqK3r/s4b0WEt3GpQtUegyj524W3VHoEqunWnjas9AlXU3NhY7RGookXNC3LLa1dm3rx5Sx1w+ner1BG2RYsWLfMOjWeccUZmz56dUaNGtf9QAAAAbWSVuobtlVdeyUYbbZQ99tgj73//+7Nw4cL84Q9/yL333psNN9ww48aNq/aIAAAArWaVCrY111wzX/ziF3PLLbfktttuy/z587PhhhvmyCOPzJgxY7LhhhtWe0QAAIBWs0oFW9euXXPBBRdUewwAAIB2sUpdwwYAANCZCDYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCCTYAAIBCrXSwjRgxInPnzl1qeUNDQ0aMGNEaMwEAAJB3EWzTpk3LggULllo+f/783H777a0yFAAAAEmXFd3woYceavn8kUceyXPPPdfyuKmpKTfddFM22mij1p0OAACgE1vhYBs6dGgqlUoqlcoyT31cY401ct5557XqcAAAAJ3ZCgfbzJkz09zcnEGDBuWee+5Jr169WtZ17do1vXv3Tm1tbZsMCQAA0BmtcLANGDAgSbJ48eLlbtPc3JxKpfLepwIAAGDlbzpSX1+fV199danls2bNys4779wqQwEAAPAugu3BBx/M1ltvnbvuuqtl2eTJkzNkyJCsv/76rTocAABAZ7bCp0S+6Z577sk3vvGNDB8+PCeccEKeeOKJ3Hjjjfmf//mfHHHEEW0xIwAAQKe00sG22mqr5dvf/nbWXHPNnHbaaenSpUumT5+eHXbYoS3mAwAA6LRW+pTIhQsX5oQTTsiZZ56Z0aNHZ4cddsjee++dG264oS3mAwAA6LRW+gjbtttum9deey3Tpk3L9ttvn+bm5px11lnZe++9c9hhh+WCCy5oizkBAAA6nZU+wrbtttvmgQceyPbbb58kqVQqOemkk3LXXXfltttua/UBAQAAOquVPsJ28cUXL3P5Nttskz/96U/veSAAAADesNJH2JLk8ssvz4477pi+fftm9uzZSZJzzjknN910U6sOBwAA0JmtdLBdeOGFOf7447Pnnntm7ty5aWpqSpL06NEj55xzTmvPBwAA0GmtdLCdd955+dGPfpSTTz45tbW1Lcu33Xbb/PnPf27V4QAAADqzlQ62mTNnZptttllqebdu3fLqq6+2ylAAAAC8i2DbZJNN8sADDyy1/Kabbsrmm2/eGjMBAACQd3GXyOOPPz5HHXVU5s+fn+bm5txzzz2ZMmVKJk6cmIsuuqgtZgQAAOiUVjrYDj/88Kyxxho55ZRT8tprr+WAAw5I3759c+6552b//fdvixkBAAA6pZUOtiQ58MADc+CBB+a1117LK6+8kt69e7f2XAAAAJ3eSl/DNmLEiMydOzdJsuaaa7bEWkNDQ0aMGNGqwwEAAHRmKx1s06ZNy4IFC5ZaPn/+/Nx+++2tMhQAAAArcUrkQw891PL5I488kueee67lcVNTU2666aZstNFGrTsdAABAJ7bCwTZ06NBUKpVUKpVlnvq4xhpr5LzzzmvV4QAAADqzFQ62mTNnprm5OYMGDco999yTXr16tazr2rVrevfundra2jYZEgAAoDNa4WAbMGBAkmTx4sVtNgwAAAD/Z6VvOgIAAED7EGwAAACFEmwAAACFEmwAAACFWulgGzRoUF566aWlls+dOzeDBg1qlaEAAAB4F8E2a9asNDU1LbW8sbExTz/9dKsMBQAAwErc1v+Xv/xly+c333xz6urqWh43NTVl6tSpGThwYKsOBwAA0JmtcLCNGjUqSVKpVHLIIYcssW611VbLwIEDc/bZZ7fqcAAAAJ3ZCgfbm2+Yvckmm+Tee+/N+uuv32ZDAQAAsBLB9qaZM2e2xRwAAAD8m5UOtgkTJrzt+lNPPfVdDwMAAMD/Welgu+6665Z4vHDhwsycOTNdunTJpptuKtjehUqX1VKprFbtMaiCpnkN1R6BKqosbq72CFTZZZM+Xu0RqKLV9vczoDPr+dcF1R6BKlq0aH4yfcW2Xelgu//++5da1tDQkPr6+nzmM59Z2d0BAACwHCv9PmzL0r1794wfPz5jxoxpjd0BAACQVgq2JJk3b17mzZvXWrsDAADo9Fb6lMjvfe97Szxubm7Os88+m8svvzyf+MQnWm0wAACAzm6lg+273/3uEo9ramrSq1evHHLIIRk9enSrDQYAANDZeR82AACAQr2na9jmzJmTOXPmtNYsAAAAvMVKB9uiRYsyZsyY1NXVZeDAgRk4cGDq6upyyimnZOHChW0xIwAAQKe00qdEHnPMMbn22mtz1llnZYcddkiS3HXXXRk3blxeeumlXHjhha0+JAAAQGe00sH2k5/8JFdeeeUSd4Tceuut079//3z+858XbAAAAK1kpU+J7NatWwYOHLjU8k022SRdu3ZtjZkAAADIuwi2o48+OqeddloaGxtbljU2Nub000/P0Ucf3arDAQAAdGYrfUrk/fffn6lTp6Zfv34ZMmRIkuTBBx/MggULsttuu2Xvvfdu2fbaa69tvUkBAAA6mZUOth49emSfffZZYln//v1bbSAAAADesNLBdumll7bFHAAAAPyblb6GbcSIEZk7d+5SyxsaGjJixIjWmAkAAIC8i2CbNm1aFixYsNTy+fPn5/bbb2+VoQAAAFiJUyIfeuihls8feeSRPPfccy2Pm5qactNNN2WjjTZq3ekAAAA6sRUOtqFDh6ZSqaRSqSzz1Mc11lgj5513XqsOBwAA0JmtcLDNnDkzzc3NGTRoUO6555706tWrZV3Xrl3Tu3fv1NbWtsmQAAAAndEKB9uAAQOSJIsXL26zYQAAAPg/K31b/8suu+xt1x988MHvehgAAAD+z0oH21e+8pUlHi9cuDCvvfZaunbtmjXXXFOwAQAAtJKVvq3/v/71ryU+Xnnllfz1r3/NRz/60UyZMqUtZgQAAOiUVjrYluV973tfzjjjjKWOvgEAAPDutUqwJUmXLl3yzDPPtNbuAAAAOr2Vvobtl7/85RKPm5ub8+yzz+b888/Pjjvu2GqDAQAAdHYrHWyjRo1a4nGlUkmvXr0yYsSInH322a01FwAAQKe30sHmfdgAAADax7u+hu3FF1/Miy++2JqzAAAA8BYrFWxz587NUUcdlfXXXz99+vRJnz59sv766+foo4/O3Llz22hEAACAzmmFT4n85z//mR122CFPP/10DjzwwGy++eZJkkceeSSTJk3K1KlT8/vf/z49e/Zss2EBAAA6kxUOtgkTJqRr166ZMWNG+vTps9S6j33sY5kwYUK++93vtvqQAAAAndEKnxL585//PN/5zneWirUk2WCDDXLWWWfluuuua9XhAAAAOrMVDrZnn302//Ef/7Hc9VtuuWWee+65VhkKAACAlQi29ddfP7NmzVru+pkzZ2bddddtjZkAAADISgTbxz/+8Zx88slZsGDBUusaGxszZsyYjBw5slWHAwAA6MxW6qYj2267bd73vvflqKOOymabbZbm5uY8+uijueCCC9LY2JjLL7+8LWcFAADoVFY42Pr165e77rorX/7ylzN69Og0NzcnSSqVSvbYY4+cf/756d+/f5sNCgAA0NmscLAlySabbJIbb7wx//rXv/L4448nSQYPHuzaNQAAgDawUsH2pp49e+bDH/5wa88CAADAW6zwTUcAAABoX4INAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUIINAACgUILtPapUKhk+fHi1xwAAADogwQYAAFAowQYAAFCoThNsl156abbbbrusvfbaWXvttbPddttl0qRJS2wzadKkVCqVpZYnybRp01KpVDJu3LglHifJ9OnTU6lUWj6W9XwAAICV1aXaA7SHY489Nuedd1422mijfPGLX0ySXHPNNTn00ENz//3359xzz13pfQ4cODBjx47N+PHjM2DAgNTX17esGzp0aCtNDgAAdGYdPthuu+22nHfeedl8881z1113pa6uLkkybty4bL/99vne976XfffdNzvttNNK7XfgwIEZN25cxo8f3/L5O2lsbExjY2PL44aGhpX6mgAAQOfS4U+JnDx5cpI3Au3NWEuSnj17ZuzYsUnSbqcwTpw4MXV1dS0f/fv3b5evCwAArJo6fLDdf//9SbLMW+/vuuuuSZIHHnigXWYZPXp05s2b1/IxZ86cdvm6AADAqqnDnxLZ0NCQmpqa9OrVa6l1ffr0SaVSabdTE7t165Zu3bq1y9cCAABWfR3+CFv37t2zePHi/OMf/1hq3QsvvJDm5uZ07949SVJT88Zfx6JFi5badt68eW07KAAAwL/p8MG2zTbbJHnjNvz/7s1lb97VsWfPnkmSp59+eqlt3zy18t/V1NSkqanpvQ8KAADwbzp8sB1yyCFJkvHjxy9x6uO8efMyfvz4Jbb50Ic+lEqlkiuvvDLz589v2fbxxx9f7q3/11133Tz11FNtNT4AANCJdfhr2Hbeeeccc8wxOe+887Lllltmn332SXNzc6655po89dRTOfbYY7PzzjsnSfr27ZvPf/7z+clPfpIPfehDGTlyZF544YVcd911GTlyZK655pql9j9ixIj87Gc/y6hRo7LNNtuktrY2n/rUp7L11lu39x8VAADoYDp8sCXJ9773vWyzzTa58MIL88Mf/jBJ8h//8R+ZMGFCDj300CW2veiii7L++uvnpz/9ab7//e/nAx/4QH74wx+mb9++ywy2N4+83XLLLbn++uuzePHi9OvXT7ABAADvWaW5ubm52kN0Vg0NDamrq8uuq+2XLpXVqj0OVdC8aGG1R6CKmj8ypNojUGXP7rBmtUegilZ71a9gnVnPvy6o9ghU0aJF83PH9PGZN29eyw0Ql6fDX8MGAACwqhJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhepS7QFImhcuSHOludpjAO2s9qEZ1R6BKuu9+vurPQJV9MLRr1d7BKpowfPrVHsEqmjRwhXPMEfYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACiXYAAAACtWl2gN0Jo2NjWlsbGx53NDQUMVpAACA0jnC1o4mTpyYurq6lo/+/ftXeyQAAKBggq0djR49OvPmzWv5mDNnTrVHAgAACuaUyHbUrVu3dOvWrdpjAAAAqwhH2AAAAAol2AAAAAol2FrJjBkz8thjj2XhwoXVHgUAAOggBFsr2W233bL55pvn6aefrvYoAABAByHYAAAACuUuka1k1qxZ1R4BAADoYBxhAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKJRgAwAAKFSXag8A0Fktfvnlao9AlXW55b5qj0AV9Xv+A9UegSq68Tc/qPYIVFHDy4vT85crtq0jbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIWqarBNmjQplUql5WP//fev5jgrbf78+UvMX6lUqj0SAADQgRRxhO3Tn/50xo4dm3333bdlWX19/VIx9O8fkyZNatn+rfE3ceLEZX6dM844Y6nnJcnAgQOX2G+3bt3Sq1evfPjDH85RRx2VO+64Y5n769KlS8aOHZuxY8dmwIAB7/nvAQAA4K26VHuAJBk1alTq6+uXue6LX/xi+vXrt8x1Q4cOXebyM888M0ceeWTWXXfdFZ6htrY2p5xySpJk0aJF+de//pU///nP+cEPfpALLrggn/zkJzN58uT07Nmz5TldunTJuHHjkiTTpk3L7NmzV/jrAQAAvJMigu3tHH744dl+++1XePtNN900M2bMyOmnn56zzz57hZ/31vh6q9mzZ+eLX/xirr/++nzmM5/JLbfckpqaIg5MAgAAHVyHK4/6+voMHjw43//+9/P3v//9Pe9vwIABuf7667P55ptn+vTpufrqq1thSgAAgHfW4YKtS5cuOf3009PY2JgxY8a0yj7XWGON/Pd//3eS5Kc//em73k9jY2MaGhqW+AAAAFie4k+JvOiii3LTTTctc93Xv/71rL766kst32+//fKd73wnV1xxRU444YRsvfXW73mO4cOHJ0nuvffed72PiRMnZvz48e95FgAAoHMoPtguvvji5a477rjjlhlslUolZ555ZkaMGJGvf/3rueGGG97zHH379k2SvPjii+96H6NHj87xxx/f8rihoSH9+/d/z7MBAAAdU/GnRN51111pbm5e5kePHj2W+7xdd901I0eOzI033pjp06e338Bvo1u3bunevfsSHwAAAMtTfLC9F2eccUZqampy4oknvud9PfPMM0mSXr16ved9AQAArIgOHWxDhgzJgQcemHvuuSdXXXXVe9rXtGnTkiTDhg1rhckAAADeWYcOtiQ57bTT0q1bt5x88slZtGjRu9rH66+/3vKebp///OdbczwAAIDl6vDBNmDAgHz5y1/O448/nkmTJq308//+97/nk5/8ZB555JHsuuuu2XvvvVt/SAAAgGUo/i6Rb3db/+233z4jR458x32cfPLJueSSSzJjxozlbrNo0aKMGzcuSdLU1JS5c+fmoYceyp133pmmpqZ8+tOfzqRJk1KpVN7VnwMAAGBlFR9sb3db/6985SsrFGzrrbdeTjrppHzjG99Y7jZNTU0t75HWtWvXdO/ePZtsskmOPPLIHHDAAdlxxx1XfngAAID3oNhgmzRp0kqdwlhfX5/6+vrlrh89enRGjx69zHWzZs1aueEAAADaQRHXsB166KGpVCrZf//9qz3KSpk/f34qlUoqlUox7/UGAAB0HFU9wjZ06NCMHTu25fGWW25ZxWlWXpcuXZaYHwAAoDVVPdiGDh1azRHeky5durTcqAQAAKC1FXFKJAAAAEsTbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIUSbAAAAIXqUu0BOrPm5uYkyaIsTJqrPAwAVVCp9gBUUU1TY7VHoIoaXl5c7RGoooZX3nj93+yBt1NpXpGtaBNPPfVU+vfvX+0xAACAKpgzZ0769ev3ttsItipavHhxnnnmmayzzjqpVDrf/2VtaGhI//79M2fOnHTv3r3a41AFvgc6N69/5+b1x/dA59bZX//m5ua8/PLL6du3b2pq3v4qNadEVlFNTc07FnVn0L179075Lyr/x/dA5+b179y8/vge6Nw68+tfV1e3Qtu56QgAAEChBBsAAEChBBtV061bt4wdOzbdunWr9ihUie+Bzs3r37l5/fE90Ll5/Vecm44AAAAUyhE2AACAQgk2AACAQgk2AACAQgk2AACAQgk2AACAQgk2ADql+vr6jBo1qt2/7qRJk9KjR4933G7cuHEZOnRoq3/9ttovAG1DsAEAABRKsAFAkuHDh+fYY4/NiSeemHXXXTcbbLBBxo0bt8Q2lUolF154YT7xiU9kjTXWyKBBg3L11Ve3rJ82bVoqlUrmzp3bsuyBBx5IpVLJrFmzMm3atBx66KGZN29eKpVKKpXKUl8jeeMo3Pjx4/Pggw+2bDdp0qQkydy5c3P44YenV69e6d69e0aMGJEHH3wwSfKPf/wjG2ywQb71rW+17Ov3v/99unbtmqlTp77tfgEoU5dqDwAApZg8eXKOP/74/OEPf8hdd92V+vr67Ljjjtljjz1athkzZkzOOOOMnHvuubn88suz//77589//nM233zzd9z/Rz7ykZxzzjk59dRT89e//jVJsvbaay+13ec+97k8/PDDuemmm/K73/0uSVJXV5ck2W+//bLGGmvkxhtvTF1dXX7wgx9kt912y9/+9rf06tUrl1xySUaNGpWPfexj+cAHPpCDDjooRx99dHbbbbe8/vrry90vAGUSbADw/2y99dYZO3ZskuR973tfzj///EydOnWJYNtvv/1y+OGHJ0lOO+20/Pa3v815552XCy644B3337Vr19TV1aVSqWSDDTZY7nZrrLFG1l577XTp0mWJ7e64447cc889eeGFF9KtW7ckyXe+8538/Oc/z9VXX53/7//7/7LnnnvmiCOOyIEHHphtt902a621ViZOnPi2+wWgXIINAP6frbfeeonHG264YV544YUllu2www5LPX7ggQfaerQkyYMPPphXXnkl66233hLLX3/99cyYMaPl8Xe+851sueWWueqqq/KnP/2pJe4AWPUINgD4f1ZbbbUlHlcqlSxevHiFn19T88al4c3NzS3LFi5c2DrDJXnllVey4YYbZtq0aUute+udJ2fMmJFnnnkmixcvzqxZs7LVVlu12gwAtC/BBgAr4e67787BBx+8xONtttkmSdKrV68kybPPPpuePXsmyVJH37p27ZqmpqZ3/DrL2u6DH/xgnnvuuXTp0iUDBw5c5vMWLFiQL3zhC/nc5z6XD3zgAzn88MPz5z//Ob17916prw9AGdwlEgBWwlVXXZVLLrkkf/vb3zJ27Njcc889Ofroo5MkgwcPTv/+/TNu3Lg8/vjj+fWvf52zzz57iecPHDgwr7zySqZOnZoXX3wxr7322jK/zsCBAzNz5sw88MADefHFF9PY2Jjdd989O+ywQ0aNGpXf/OY3mTVrVn7/+9/n5JNPzh//+Mckycknn5x58+ble9/7Xk466aS8//3vz2GHHfa2+wWgXIINAFbC+PHjc+WVV2brrbfOZZddlilTpmSLLbZI8sYplVOmTMljjz2WrbfeOmeeeWa++c1vLvH8j3zkI/nSl76Uz33uc+nVq1fOOuusZX6dffbZJyNHjsyuu+6aXr16ZcqUKalUKrnhhhuy884759BDD8373//+7L///pk9e3b69OmTadOm5Zxzzsnll1+e7t27p6amJpdffnluv/32XHjhhcvdLwDlqjS/9UR7AGC5KpVKrrvuuowaNaraowDQSTjCBgAAUCjBBgAAUCh3iQSAFeQqAgDamyNsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhRJsAAAAhfr/AQHsu+d6OZUIAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%time\n",
" # Try to find out.'\n",
"model.plot_attention('Tratar de descubrir.')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rA3xI3NzrRJt"
},
"source": [
"The short sentences often work well, but if the input is too long the model literally loses focus and stops providing reasonable predictions. There are two main reasons for this:\n",
"\n",
"1. The model was trained with teacher-forcing feeding the correct token at each step, regardless of the model's predictions. The model could be made more robust if it were sometimes fed its own predictions.\n",
"2. The model only has access to its previous output through the RNN state. If the RNN state looses track of where it was in the context sequence there's no way for the model to recover. [Transformers](transformer.ipynb) improve on this by letting the decoder look at what it has output so far."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vtz6QBoGWqT2"
},
"source": [
"The raw data is sorted by length, so try translating the longest sequence:"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:11.622335Z",
"iopub.status.busy": "2023-12-07T12:24:11.622091Z",
"iopub.status.idle": "2023-12-07T12:24:11.626087Z",
"shell.execute_reply": "2023-12-07T12:24:11.625436Z"
},
"id": "-FUHFLEvSMbG"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Expected output:\n",
" If you want to sound like a native speaker, you must be willing to\n",
"practice saying the same sentence over and over in the same way that\n",
"banjo players practice the same phrase over and over until they can\n",
"play it correctly and at the desired tempo.\n"
]
}
],
"source": [
"long_text = context_raw[-1]\n",
"\n",
"import textwrap\n",
"print('Expected output:\\n', '\\n'.join(textwrap.wrap(target_raw[-1])))"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:11.629569Z",
"iopub.status.busy": "2023-12-07T12:24:11.628778Z",
"iopub.status.idle": "2023-12-07T12:24:12.814050Z",
"shell.execute_reply": "2023-12-07T12:24:12.813276Z"
},
"id": "lDa_8NaN_RUy"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/tmpfs/tmp/ipykernel_16852/3355722706.py:23: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_xticklabels([''] + context, fontdict=fontdict, rotation=90)\n",
"/tmpfs/tmp/ipykernel_16852/3355722706.py:24: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.\n",
" ax.set_yticklabels([''] + output, fontdict=fontdict)\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA4cAAAH4CAYAAAD5BpirAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAADr3UlEQVR4nOzdd1wUx/8/8NceXZFerBQVSyxgQWwI2CtiNyYo2DX2XhIFY40l0agxsaGxRaMfayxBKdLUqBgVjRVEsSNgAwTevz/8sV+OO9rewYG+n4/HPfR2d3bm2L25nd2Z9whERGCMMcYYY4wx9lmTaboAjDHGGGOMMcY0jxuHjDHGGGOMMca4ccgYY4wxxhhjjBuHjDHGGGOMMcbAjUPGGGOMMcYYY+DGIWOMMcYYY4wxcOOQMcYYY4wxxhi4ccgYY4wxxhhjDNw4ZIwxxhhjjDEGbhwyxhhjjDHGGAM3DhljjDHGGGOMgRuHjDHGGGOMMcYAaGu6AIwxxhhjjLHS6c2bN7h16xbevn0LV1dXTReHFTN+csgYY4wxxhiTExsbi549e8LU1BTOzs7w8PAQ14WHh+OLL75AcHCw5grIigU3DhljjDHGGGOiBw8eoHnz5vjrr7/Qs2dPtGjRAkQkrndxccGLFy+we/duDZaSFQduHDLGGGOMMcZE8+fPx6tXrxASEoI///wTHTp0kFuvra0NV1dXhIeHa6iErLhw45AxxhhjjDEmOnnyJHr16oWWLVvmuY2trS0ePXpUgqViJYEbh4wxxhhjjDFRYmIi7Ozs8t2GiJCWllYyBWIlhhuHjDHGGGOMMZG1tTVu376d7zZXr16FjY1NCZWIlRRuHDLGGGOMMcZEHTp0wNGjR/Hvv/8qXX/27FmcOXMGXbt2LeGSseImUM7QQ4wxxhhjjLHPWmxsLJycnAAA06dPx82bN7Fr1y4cPXoUERERWLVqFcqXL48rV66gUqVKmi0sUytuHDLGGGOMMcbknDt3DgMHDkRcXBwEQQARif/a2Njgzz//RNOmTTVdTKZm3DhkjDHGGGOMKcjIyMCRI0dw7tw5JCYmwsjICC4uLujZsyd0dXU1XTxWDLhxyBhjjDHGGGOMA9IwxhhjjDHG/k/btm2xffv2fLfZsWMH2rZtW0IlYiWFG4eMMcYYY4wxUXBwMGJjY/PdJi4uDiEhISVTIFZiuHHIGGOMMcYYK5K3b99CR0dH08Vgaqat6QIwxhhjjDHGNOvBgwdy75OSkhSWAUBmZibi4+Oxf/9+2NnZlVDpWEnhgDSMMcYYY4x95mQyGQRBKPT2RITly5dj6tSpxVgqVtL4ySFjjDHGGGOfucGDB4vzGG7fvh2Ojo5wcnJS2E5LSwtmZmZo27YtOnfuXPIFZcWKnxwyxhhjjDHGRPb29pg8eTImTJig6aKwEsaNQ8YYY4wxxhhjHK2UMcYYY4wxxhg3DhljjDHGGGO5BAYGomvXrrC0tISOjg60tLQUXtraHL7kU8NHlDHGGGOMMSbav38/BgwYgKysLNja2qJOnTrcEPxM8JhDxhhjjDHGmMjR0RH37t3DoUOH0LZtW00Xh5Ug7lbKGGOMMcYYE/33338YOHAgNww/Q9w4ZIwxxhhjjInMzc1Rrlw5TReDaQA3DhljjDHGGGOivn37IjAwEBkZGZouCithPOaQMcYYY4wxJnr79i06duyIihUr4scff4SNjY2mi8RKCDcOGWOMMcYYY6Lq1avjw4cPSEhIAACYmJjA2NhYYTtBEHD37t2SLh4rRhyTljHGGGOMMSbKysqCtra23BNDZc+T+BnTp4efHDLGGGOMMcYY44A0jDHGGGOMMca4WyljjDHGGGMsDzExMbh58ybevn0Lb29vTReHFTN+csgYY4wxxhiTc+HCBTg5OaFBgwbo168ffHx8xHWhoaEoV64cDh8+rLkCsmLBjUPGGGOMMcaY6Pr162jbti3u37+PyZMno0uXLnLrXV1dYWFhgX379mmohKy4cOOQMcYYY4wxJpo/fz4A4OLFi1ixYgWcnZ3l1guCgBYtWuDChQuaKB4rRtw4ZIwxxhhjjIlCQkLQp08f1KxZM89tbGxs8Pjx4xIsFSsJ3DhkjDHGGGOMiV6/fg0rK6t8t3n//j0yMzNLqESspHDjkDHGGGOMMSaqVq0arl69mu82ly5dQo0aNUqoRKykcOOQMcYYY4wxJurevTtOnTqFwMBApev37t2LqKgoeHl5lWzBWLETiIg0XQjGGGOMMcZY6fD8+XM0btwYT58+xZAhQ/DkyRP89ddf+PnnnxEZGYndu3fDxsYGly9fhrGxsaaLy9SIG4eMMcYYY4wxOffu3YO3tzciIyMV1rm4uGD37t2ws7Mr+YKxYsWNQ8YYY4wxxphS0dHRiIqKQmJiIoyMjODi4qIwtQX7dHDjkDHGGGOMMcYYB6RhjDHGGGOMMQZoa7oAjDHGGGOMsdLl9evX2Lx5M65cuYKEhAR8+PBBYRtBEHD69GkNlI4VF24cMsYYY4wxxkQXLlxAly5d8OrVK+Q3Ak0QhBIsFSsJ3K2UMcYYY4wxJpo4cSKSkpKwdOlSPHjwAB8+fEBWVpbCKzMzU9NFZWrGAWkYY4wxxhhjIgMDA/Tp0wc7duzQdFFYCeNupYyxYpeRkYH//vsPKSkpMDIyQu3ataGtzdUPY4wxVhqZmZnB0tJS08VgGsDdShljxSYxMREjRoyAsbExGjZsiNatW6Nhw4YwMTHByJEj8fLlS00XkTHGGGO5eHl54cyZM8jKytJ0UVgJ426ljLFikZiYiObNm+POnTswMzND06ZNUalSJTx58gT//PMPXr58CQcHB0RGRsLMzEzTxWWMMcbY/5eSkoK2bdvCwcEBK1asQJUqVTRdJFZCuHHIGCsWkydPxurVqzF9+nTMmzcP5cuXF9e9e/cO33//PZYtW4bJkydj5cqVGiwpY4wxxnK7dOkS2rdvj+TkZJiamsLIyEhhG0EQcPfuXQ2UjhUXbhwyxopF9erVYWdnhzNnzuS5Tdu2bREbG4t79+6VYMkYY4wxlp/Tp0+jR48eSE1NhY6ODqysrPKMFXD//v0SLl3pEhsbi507dyI6OlqMrdCoUSMMGjQIdnZ2mi5ekXFECMZYsUhISMCXX36Z7zYtWrRARERECZWIMcYYY4Uxc+ZMEBH++OMP9O3bl+czzMPq1asxY8YMZGRkyM0HuX//fvj7++OHH37AxIkTNVjCouOANIyxYmFsbIy4uLh8t4mLi4OxsXEJlYgxxhhjhRETE4Ovv/4a/fr144ZhHo4ePYrJkyfD2NgYCxcuREREBO7fv4/IyEgsXrwYxsbGmDJlCo4dO6bpohYJNw4ZY8XCzc0N+/btQ2BgoNL1p0+fxr59++Du7q7WfKtXr45vvvlGrftkjDHGPieWlpYwMDDQdDFKtVWrVsHMzAyXLl3C7Nmz0bx5c9ja2sLFxQWzZs3CxYsXYWpqilWrVmm6qEXC3UoZY8Vi/vz5OHbsGDp16oSuXbvCzc0N1tbWePr0KYKDg3H8+HGUK1cO8+bNU2u+L168UDponjHGGGOF89VXX2Hfvn14//49NxLzcOnSJXz11VeoWrWq0vXVqlVD//79sWvXrhIumWq4cciYii5fvozdu3fj5s2bePfunfikLC4uDufOnUP79u0/y6ka6tWrh5MnT8LHxwfHjh3DsWPHIAiC2Ce/Ro0aCAgIQL169dSab8OGDXHr1i217pMxxhj7nPj5+eHGjRvo3LkzFi9eDEdHRxgaGmq6WKVKenq6XCR2ZQwNDZGenl5CJVIPjlbKmApmzJiBlStXig0eQRCQmZkJ4GP0qpo1a2LlypVlbjCyOhERwsPDcfnyZbkoXq1atSqWcQxHjhxBnz59cPLkSXh4eKh9/4wxxtinTktLC8DH3/D8fqsFQUBGRkZJFatUcXJyQlpaGq5evao0kmtGRgYaNmwIXV1dREdHl3wBJeInh4xJtHXrVqxYsQI9evTAokWLsHv3bixdulRcb2dnh2bNmuHw4cOfdeNQEAS0bt0arVu3LpH8Xr16hY4dO6Jjx47w8vKCs7MzrK2tlf64DR48uETKxBhjjJUlrq6uHIimAIMHD8a0adPQqVMn/PDDD2jSpIm47p9//sHs2bPx33//YcWKFRosZdHxk0PGJHJ2dsa7d+9w5coVaGtrw9/fHwsWLBCfHALA0KFDERgYiAcPHmiwpJpRvXp1TJo0CRMmTMhzm3Xr1mHlypVqnedQJpPJdV/NlvNHLvtOaM5jle3t27c4ePCg3HxFTk5O8PLyKrD7CGOMMcY+D5mZmejTpw8OHz4MQRBQrlw5WFlZ4dmzZ3j37h2ICD179sT+/fshk5WdGKD85JAxiWJiYjBixIg8J4UFAGtrazx79qwES1V6xMbGIikpKd9tkpKSCpzuoqi2bt0qOe3+/fsxcuRIJCUlyTUuBUGAiYkJNm7ciN69e6ujmIwxxhgrw7S0tHDw4EFs374d27ZtQ3R0NB48eAAjIyO4uLhgyJAh8Pb21nQxi4wbh4xJpK2tXeAg44SEBB7AnY/k5GTo6empdZ9DhgyRlC4iIgIDBw6ElpYWhg8fDg8PD1SqVAlPnjxBUFAQtm3bhoEDByIkJAQtWrRQa5kZY4wxVjYNHjz4kxqmwo1DxiRq0KABzpw5g8zMTHHgdk7ZkUtz9kH/1IWGhsq9j42NVVgGfOyKER8fj507d6JWrVolVbx8LV68GHp6eggPD4ejo6PcugEDBmDs2LFo2bIlFi9ejCNHjmiolIwxxljJyMzMxN69exEYGIiEhASkpaUpbCMIAk6fPq2B0rHiwo1DxiQaOnQohg8fjtGjR2Pt2rVy61JSUjB8+HA8efIEq1ev1lAJS567u7s4tk8QBGzbtg3btm1Tum32uL+cQXzULTMzEy9evFD6gwYANjY24v8jIyMxYMAAhYZhtoYNG6J///44dOhQsZSVMVY2FKVeYaysevv2LTp27IioqCjx9zr3cIuCIpl+alSJH1GW6gVuHDImUXawmc2bN+OPP/6AiYkJAKBZs2a4ceMG3r59Cx8fH/Tt21ezBS1B8+bNE38wFixYADc3N7i7uytsp6WlBTMzM3h4eKBu3bpqL8fFixcxZ84chIaG5tn1N3f47Xfv3sHa2jrf/VpbW+Pdu3dqLStjrGyQUq8wVlYtXLgQkZGRWLBgAcaOHQsLCwv4+flh1KhRCA0NxZw5c9C4cWPs3LlT00UtMXZ2dpIaw2WtXuBopYypaOPGjVi7di2uXbsm3lWrW7cuJkyYgFGjRmm4dJrj4eEBX1/fEu+HHx0djZYtW0JbWxseHh44cuQIHB0dUbFiRVy6dAnPnz+Hu7s7bG1t5YLX1K1bF4aGhrhw4UKe+27WrBlev36NGzdulMRHYYyVElLrFcbKqtq1a8Pc3BwREREAPkYC9/Pzw7x58wAADx8+hKOjI6ZNm4bZs2drsqglxsfHR6FxeO/ePZw9exYmJiZwcnKCtbU1nj59iujoaCQlJcHV1RXVq1cvW/UCMcbU4t27d/To0SN6/fq1povyWevduzcZGBhQTEwMEREJgkD+/v5E9PEYjRkzhiwtLen+/fty6ebNm0eCINDgwYPp0aNHcusSEhJoyJAhJJPJaP78+SXxMRhjpYjUeoWxskpfX58mT54svtfS0qI5c+bIbePt7U21a9cu6aKVGteuXSNjY2OaO3cuvXnzRm7dmzdvaPbs2WRiYkLXr1/XUAml4SeHn5C2bduqvA8fH59PKuJScQoNDYWdnV2+/cjj4+Nx//59tGnTpgRLVjrEx8fj9u3baN68OcqVKwcAyMrKwvLly3H48GEYGBhg8uTJ6Natm1rztba2hoeHB/bs2QPg493O+fPnY/78+WIZGjdujC+++AK7du0S07179w4eHh64cOECdHV1UbNmTfEO4J07d5Ceno5mzZohKCgIBgYGai0zY7nFx8fnGQACwGdZp2iS1HqFsbLKwsICPj4+4gTu5ubm6NmzJ7Zs2SJuM2PGDKxdu/azHW7RrVs3fPjwAadOncpzmw4dOkBfX79MBbLjMYefkODgYJXSC4KgdHwYU87DwwPz588Xu1gos337dsybN0/pZOufuu+++w5HjhzBkydPxGWLFi0SL6YAICQkBBEREXB2dlZbvsnJyahevbr4XkdHB2/evBHfy2QyuLu7Y/fu3XLpypUrh9DQUCxbtgzbt29HTEwMYmJiAADVq1fHkCFDMGPGDLVPvcFYTkeOHMH06dNx+/btfLf7HOsUTZJarzBWVtna2soFYKlfvz7OnDmDtLQ06OnpgYhw+vRpVKpUSYOl1Kzw8HCMGzcu322aNWuGdevWlVCJ1IMbh5+YnP3Bi0omk6m5NP8nLS0NMpkMOjo6xZZHSSvMQ/esrKzPKpJXTuHh4Wjfvr14zIkIa9euRZ06dXDq1Ck8efIE7du3x/Lly7F371615WtlZYVXr16J7ytWrKhwoZ2amqr0Tqeenh7mzZuHefPm4fXr10hJSYGRkREqVKigtvIxlpfg4GD06tULFStWxLhx4/Dzzz/Dzc0NderUQVhYGK5fv47u3bt/VtPjlBaq1CuMqdP58+dx4cIFJCUlKb1JJAgCvvvuO5XzadeuHbZu3YqMjAxoa2tjyJAhGD58OFq0aIF27dohIiIC0dHRmDp1qsp5lVVZWVm4c+dOvtvcvn27UNeLpYom+7Qy9co5BkIT6UNCQui7776jV69eictevHhBnTt3Jm1tbTIwMKCZM2dK3n9pU5i/l7e3N5mZmZVQiUoXIyMjmj59uvj+0qVLJAgCrV+/Xlw2ZMgQsrOzU2u+nTp1Ig8PD/H9l19+SQYGBhQREUFERDExMWRqakpNmzZVa76MqapTp05kbGxMT548ISLFOmbx4sVUrlw5unz5soZK+PnieoVp2suXL8nV1ZVkMhkJgpDnSyaTqSW/W7du0dKlSykhIUFcNm7cOLn8+/btS2/fvlVLfmVRt27dSFtbm3bv3q10/a5du0hLS4u6d+9ewiVTDT85/IQcP34cNWvW1Fj6FStWICYmBgsWLBCXTZ06FSdPnkTNmjXx5s0bLF++HI0bN0b//v0l56NJQ4cOlXt/8OBBxMbGKmyXPcl7aGgounTpUkKlK12ysrKQlZUlvg8ODoYgCHJjY6tUqSLX7VQdunXrhsmTJ+Px48eoVKkSZs6cif/9739o3bo1zMzM8OrVK2RlZWHOnDlqzZcxVV24cAFeXl5yU6rk/A7Nnj0bx44dw7x583D48GFNFPGzxfUK07QpU6YgLCwM7u7uGDJkCKpWrQpt7eK7jHdwcMDMmTPllv3888+YN28e7t27B1tbW1SsWLHY8i8LfvjhB5w9exZfffUVli1bhtatW8PKygrPnj1DWFgY/v33X1SoUAHLli3TdFGLhAPSMLWpVq0a2rVrh4CAAAAfu9iYmZmhTZs2OHHiBF6/fo2GDRvCzs4OQUFBmi2sRDm73uaeEDY3QRDg7OyMHTt2qNToLqvq1asHc3NzhIaGAgDc3Nxw9+5dPHz4UNxm2LBh+Ouvv/D48WO15fvhwwckJibC1NQUurq6AICIiAgsWrRI/EEbP348evTo8VnMV8TKDgMDA0yZMgWLFi0S348ZMwarVq0St5k2bRq2bt2Kly9faqqYavfo0SNER0eL3bidnJxQpUoVTRdLTmHrFXUH2GIsm4WFBWrWrInIyMgSGa7y4MEDmJiYwMjIKM9tXr9+jVevXpWpCd7zk5GRgf/++0+si2rXrl1gA/zatWsYN26ceK2TU5s2bbBu3TrUq1evuIpcLPjJ4SemevXqmDRpEiZMmFDieb98+VLuBz0yMhKpqanw9fUFAFSoUAHdu3fH/v37S7xs6nL//n0AH8fPZf+tJ06cqLCdlpYWTE1NUb58+ZIuYqnRp08fLFq0CH379oW+vj7CwsIUBm7HxMTIBXlQBx0dHYXJ7Fu2bIljx47JLWvTpo3CD+yrV6/w77//QktLC9WqVROjlcbHxyMzMxMNGzaEqampWsvLWLaKFSvi+fPn4vsqVarg+vXrctu8fPnykwlGc+fOHYwZMwZnzpxRWNeuXTusX7++1NxYK2y9wlhxef/+vdLfreJib28PPz+/fMcvrlmz5pMIupeYmIiZM2di165dSE1NFZcbGBhg0KBBWLJkCczNzZWmrV+/PoKDgxEfH48rV64gOTkZxsbGcHR0RLVq1UrqI6gVNw4/MbGxsUhKStJI3gYGBnj9+rX4PigoCIIgwM3NTVxmaGgoN6i/rLG1tRX/v3XrVjRq1EhuGfs/06ZNw6lTp3DgwAEAQMOGDeHn5yeuj4uLw/nz5zFr1iyNlC93dN+HDx+iVatWGDRoEBYvXix3J/TBgweYPXs2wsPDcfTo0RIuKftcODo64tq1a+J7Dw8PbNu2Dbt374anpyfCwsKwd+/eTyIgTXx8PFq3bo1nz56hTp06aNOmDSpVqoQnT54gNDQUgYGBcHV1xfnz58vsBRZj6uTk5KR0GEtxIaICA6l8Cp0PExMT0bx5c9y5cwdmZmZwdXUV66J//vkHmzZtQkhICCIjI2FmZpbnfqpVq/bp1FUaHO/IioGqQWVU0axZM3JwcKDU1FRKS0sjBwcHql+/vtw2gwYNIhsbG42Uj2nG1atX6erVq5SRkSG3PDY2lg4ePEgPHz5Ua34BAQHUuHFjhYnssz169IgaN25MO3fulFs+YMAAcnFxyXffLi4uNHDgQLWVlbGcNm/eTAYGBhQbG0tERPfu3SNTU1OSyWTiS1dXl0JCQjRcUtUNGzaMBEGgX375hbKyshTWb9iwgQRBoOHDh2ugdIqk1iuMqcvJkydJT0+PIiMjSyS/wlxPjho1ioyNjUukPMVl0qRJJAgCzZgxQ2Ei+7dv39KsWbNIEASaMmWKhkpY8njM4SdGJpOpNJ2FKjZt2oSRI0eiSpUq0NXVRWxsLJYvX44pU6aI29SpUwf29vY4fvx4iZevuJRUWGlWOB4eHkhNTUVkZGSe27Ru3Rp6eno4ffq0uMzCwgKjRo0Sx3spM2fOHGzcuFGu6x9jxenu3btYtWqVOK5t9OjRcHJy0nSxVFatWjU0btwYhw4dynObnj174uLFi3LjlDVFar2iLlLGQrFPy/bt23Ho0CEcPXoUX331FRo3bpzneMDBgwdLyiNnQEE/Pz+4u7srnf86O+jenj170Lx5c6Vdw8uK6tWrw87OLt/P0LZtW8TGxuLevXsK6zIzM7F3714EBgYiISEBaWlpCtsIglAs9UJx4ZrlE6SpefWGDRuG27dvY/PmzXj//j3GjBmDSZMmiesjIyNx69YtDB8+XCPlU7fExER4eXkhPDy8wMA0n3Pj8MmTJzhw4ABu3ryJd+/eYdOmTQCA58+f4/79+2jQoAEMDAzUll9MTAz69OmT7zZOTk4KY19TU1MLDIyTkJCA9+/fq1xGZR4+fIigoKB8f1w+5/OoLFLHMa1Ro0aZm0C5MJ49e4b69evnu039+vVx4sSJEipR/qTWK6pSZSwU+7T4+PiIgfACAgIQEBCgcL1HRBAEQXLjMOfQD0EQEBwcrDAEI6fKlSuXuUicuSUkJODLL7/Md5sWLVogIiJCYfnbt2/RsWNHREVFiX/7nNeD2e/L2nzX3Dj8BP3444/YunVrobcXBAF3795VOV9BELBs2bI8K4omTZrg1atXn0yQlpIOK10WrV+/HlOnThUvjAVBEBuHz549Q4sWLbBhwwaMGDFCbXkmJycXGDTGyMhIYexrkyZNsGfPHowYMQItWrRQSBMREYE//vgDzZs3V1tZs02fPh2rV6+We/Kc8wcl+//cOCw7pBxTLS0tDBw4EDt37izx8pY0S0tLxMTE5LtNTEwMLC0tS6hE+ZNar6hCXWOh2KehKNd1UmVHkicitG3bFj4+PhgyZIjCdlpaWjAzM0OdOnXkoriXRcbGxoiLi8t3m7i4OBgbGyssX7hwISIjI7FgwQKMHTsWFhYW8PPzw6hRoxAaGoo5c+agcePGZa9O10hnVlZs8psYNb8XKzpzc3NycXFROl6GER0+fJgEQSBnZ2c6cuQIjR07VmFyXicnJ+ratata87W3ty9wn127dlUY+3r27FnS1dUlLS0t8vLyohUrVtD27dtpxYoV1LNnT9LS0iI9PT0KCwtTa3l/++03EgSBOnbsSAcOHCBBEMjX15f27NlDY8eOJV1dXerfvz8FBwerNV9WfKQeUxMTE5o5c6aGSl2yhg4dSjKZjDZt2qR0/ebNm0lLS4uGDRtWwiVTTmq9ogoeC8U0yc/P77P43enXrx/p6urS33//rXR9YGCgWGfnVqtWLWrRooX4Pvc4zfj4eDIzM6PFixerv+DFiBuHnxhNBqTJduDAAerXrx81aNCAatSoIS6/ceMGLVu2TO0BSDSlXLlyNH36dE0Xo9RydXUlW1tb8aLGz89PoXHo7e1NdnZ2as13xIgRpKWlRadOnVK6/uTJkySTyWjEiBEK6wIDA6l69eriTROZTCb+v3r16hQYGKjWshIRNW3alOzt7SkzM5OIFL/DwcHBpKOjQ/v27VN73qx4SD2mHTt2pM6dO5doWTUlLi6OLC0tSSaTUf369embb76hBQsW0DfffEMNGjQgmUxGVlZW9ODBA00XlYhUq1eksre3Jw8Pj3y38fDwIHt7e4XlO3bsoPbt25OFhQXp6uqShYUFdejQgQPmMJbLtWvXqFy5ciSTyah79+60fPly2r59Oy1fvpy6detGMpmMDA0N6dq1awpp9fX1afLkyeJ7LS0tmjNnjtw23t7eVLt27WL/HOrEjcNPjCYbh5mZmdS/f38xql758uXlGgNPnjwhbW3tMncHJS8tW7akfv36SU7/5s0b2rFjB02bNo1GjhxJ06ZNox07dijcIS6rKlSoQGPHjhXfK2sczpo1iwwMDNSa771798jIyIi0tbXJ19eXdu3aRSEhIbRr1y7y8fEhbW1tMjExoXv37ilNn5WVRaGhofTzzz/TwoUL6eeff6bQ0NBie0JsaGhIY8aMEd8LgkDfffed3DZeXl4FRlJlpYfUYxoREUG6urq0bdu2Eimnpt26dYs8PDyU9mZp27Yt/ffff5ouokjVekUKPT09hQvN3ObMmUN6enri+4yMDOrdu7d4Y8vAwIDs7OzIwMBAvOHVu3dv8cYFK5syMjLoyZMnFBcXp/SlDp9ThN6zZ89SjRo1lN4YrlmzZp49hszNzWnq1KniezMzM/L19ZXbZvr06Wq/ziluPECKqc2PP/6Iffv2YfTo0Vi6dClWrVqF77//XlxvbW0NV1dXHDt2DLNnz9ZgSdVj/vz58PT0RFRUVJHHoe3fvx8jR45EUlKSwuBlExMTbNy4Eb1791Z3kUtUVlYWdHR08t3m2bNn0NPTU2u+9vb2OHbsGAYOHIiAgABs27ZNXEdEqFq1Kvbu3Qt7e3ul6QVBgKurK1xdXdVarvyYmJiI/y9fvjxevnwpt7527doIDAwssfIw1Uk5pn///Tfc3d3h6+uLn3/+Gc7OzrC2tlYIZvCpjD91cHDAmTNnEB8fj+joaDESp5OTU6mbL0zVekUKKWOh1qxZg//9739o3bo1li1bJjd+OioqCjNnzsTBgwfx888/Y+LEiWorKysZFy9exJw5cxAaGor09HSl2wiCgIyMDJXzCggIgK6uLipXrqx0feXKlWFgYIDNmzdj0KBBKuenSa1bt8bt27cRHh6Oy5cvi3VRo0aN0KpVqzwDytja2uLBgwfi+/r16+PMmTNIS0uDnp4eiAinT59GpUqVSuqjqAU3DpnaBAQEwNnZGevXrwegPGpqzZo1cezYsZIuWrF48uQJunXrBjc3tyKFlY6IiMDAgQOhpaWF4cOHw8PDQwwyEBQUhG3btmHgwIEICQlRGhhFiqtXr+LChQvo27evWMb3799jypQpOHz4MAwMDDBt2jSMHj1aLfkBHy9+z549m+f6jIwMhIaGokGDBmrLM1vr1q1x7949HDp0COfPn0dycjJMTEzQrFkzeHp6QldXV+15SlWlShW5UP01atTAuXPn5La5du3aJxPI6XMg9ZjmjBR48eJFXLx4Uen+P5XGYbayMnl0Sdcrbm5u2LdvH3x8fNC+fXuF9adPn8a+ffvg5eUlLtu2bRtq1aqF06dPK9yca968OQIDA9GwYUNs3bqVG4dlTHR0NFxdXaGtrY2OHTviyJEjcHR0RMWKFXHp0iU8f/4c7u7usLW1VUt+morQqymCIKB169Zo3bp1odO0a9cOW7duRUZGBrS1tTFkyBAMHz4cLVq0QLt27RAREYHo6GhMnTq1GEteDDT74JKpW1G7lWZmZtLWrVvVkre+vr7c4/W8uhHm7AJTluXuepD9Pucre1lO3bp1o/Lly1N0dLTS/V65coXKly9P3bt3V1tZ+/fvT5UrV5brGpkd7KBChQqkq6tLMpksz/E0UqxcuZIEQSA/Pz8ikj8fMjIyaOLEiSSTyWjjxo1qy1NVz549o2XLllHfvn2pffv25OHhofBq27atWvMcOnQoOTg4iO/nzZtHMpmMRo4cSUePHqVZs2aRTCZTqQszK1lSj2lwcHChX2VdfHw8HTp0iF69eqV0fWJiIh06dOiTGaMuhZSxUAYGBgWOhS+L3dwYUe/evcnAwIBiYmKISP567927dzRmzBiytLSk+/fvqyW/wnRrnj179idzTUdE9OLFCzp9+jQdOHCATp8+TS9evMh3+1u3btHSpUspISFBXDZu3Di5a8O+ffvS27dvi7voasWNw09MbGwsJSUlFWrbHTt2UK1atRQaL1KZmprS8OHDxffKGodffvklWVtbqyU/TQsICCj0KyczMzMaOnRovvv29fUlMzMztZXVzs6OvvrqK/H9hw8fyNjYmFxcXOj9+/f06NEjsra2pm7duqktz/T0dHJ3dyeZTEYODg5ikIl+/fqRvb09CYJAnTp1KjXRXq9cuULm5uYKDf7cL3V9X7IFBQVR586dKTY2loiIXr9+TY0bN5a7+WBvby+uZ6UfH9OCjRo1iszMzCg1NVXp+tTUVDI3N6dvvvmmhEtWuhR1LJSxsbHcWG9lxo4dS8bGxsVYalYcrKysaMCAAeL7nDdfiT7e7Hd0dKQvv/xSLflpIkKvpty/f588PT1JS0tL7ga/lpYW9ezZs8gN7mfPnlFUVBQ9fvy4eApczLhb6SfG1tYWL1++hL+/Py5evAhtbW24urpi9OjR4kTjhw8fxuzZs3Hz5k0AUNvYtkaNGuHkyZNITU2Fvr6+wvrExEScOHECbdq0UVgXGhoqjjUpK5TN/VMY7969g7W1db7bWFtb4927d5L2r8zz58/lum1duHABKSkpGD16NPT19VG5cmX07NkTf/31l9ry1NHRwcmTJ+Hv748NGzaI83/9+eefMDIywsyZM+Hv76/y5LDbt28HAPTq1QsVKlQQ3xdGzu6+U6dORWJiIr799lsMGzYMVatWhZaWlkplKwx3d3e4u7uL7w0NDREVFYVDhw7h7t27sLW1RY8ePbhbaRnCx7RgZ86cQceOHfMcc6ynp4eOHTtqbKytuuoVVRV1LFSjRo2wd+9ezJ07V+lYscePH2Pv3r1o3Lix2srISkZycjKqV68uvtfR0cGbN2/E9zKZDO7u7ti9e7da8mvfvj22bNmCv//+Gx06dFBYf+rUKZw4cQLDhg1TS36acvfuXbRq1QrPnj2Dg4MDWrVqBWtrazx9+hQRERE4fPgwoqKiEBERIff3B4AFCxbA3t4e3t7ecsstLS3VPkerOq5H5s+fj3nz5hW8oaZbp0y9njx5QjY2NnJ3F2UyGbm6ulJ6ejoNHjxYvCPSu3dvunLlitryPnjwIAmCQF27dqX4+Hi5J4d37tyhNm3akEwmo9OnTyuklclkctH9PmV16tShpk2b5ruNs7Mz1alTR215Ghsby4VbXrx4MclkMrknF7NnzyZ9fX215ZlTVlYW3bhxg8LDw+nq1auUkZGhtn1nn+PZkQ2Vde8tTHff8uXLU69evdRWLsak+PDhA61atYqcnZ2pQoUKpKWlJa67fPkyjRkzplRF8ZSqXLlyNGvWrHy3mTVrFhkaGpZQieSpq14padnzy1aqVIlWrFhBFy5coAcPHtCFCxdo+fLlVKlSJZLJZHTkyBGNlpMVXbVq1Wj06NHiexsbG+rZs6fcNqNGjVLbd0YTEXo1oU+fPiSTyejXX39V6MmUlZVFGzZsIJlMRn379lVIq6OjU2JTmgmCQHZ2duTu7l7kl5ubW5GGnfGTw0/MokWLEB8fj65du8LHxwcAsGXLFpw8eRLt2rVDWFgY2rRpg9WrV8PR0VGteffs2RMzZ87EsmXLYGtrK94Vt7KywsuXL0FE+O6779C2bVuFtFZWVkqfNhZVeHi4QtS7Vq1aqbzfgmRmZuLFixdIS0tTut7Gxkb8f//+/fH9999jyJAhWLJkidzd3cePH2P27Nm4ePGiWgNO2NnZISgoSHy/b98+2Nvbyw1cf/ToEczNzdWWZ06CIKBOnTrFsu8tW7ZAEAQxGtjWrVsl7UdXVxc1atRQZ9EYK5L379+jY8eOiIiIgIWFBYyMjPD27Vtxvb29PbZu3QozMzMsXLhQgyVVna6uLlJSUvLdJiUlReWeBcqEh4cjICBA7reiUaNGGDx4sBiMQl31Sknr0aMHVqxYgVmzZmHGjBly64gI2traWLFiBbp3766hEjKpvvjiC/z333/i+1atWuHgwYOIjIxEixYtcOPGDezdu1dtv7WaiNCrCadPn4anpydGjhypsE4QBIwaNQp//fWX0l4MNjY2SEpKKoFSfuTr61u4J39KyGSyQm8rEOWIo8/KvNq1a0NHRwfXrl0TlxER6tevj5s3b8Lb2xsBAQHFWoa///4ba9euxblz55CYmAgjIyO4uLhgwoQJ6NSpk9I0gwcPxrVr13Dx4kVJFwMRERHw9fXFnTt3AHz8zNn7cXBwwNatW9UW+TMnKWGl3717Bw8PD1y4cAG6urqoWbOm2IXhzp07SE9PR7NmzRAUFCR2BVbVypUrMX36dDg7O0NPTw/h4eGYO3cuFixYIG7TqFEjmJiYyDUiPye9evVCUlKSRj7/69evsXnzZly5cgUJCQn48OGDwjaCIOD06dMlXjYmjZRj+t1332HRokVYunQppk+fDn9/f3z//ffIzMwUt+ncuTNevnyJCxculMjnKC5t2rRBbGwsbt++rbRraWpqKhwcHGBjY4Pw8HC15Tt58mSsWbNGnEJIEAS5/0+cOBGrVq1SW36qklo33Lt3Dzt37lRoAA8aNEiha9znLD4+HgsXLkRgYCASEhKU/o6ra2oIVf3888+YPHky4uPjUalSJVy5cgXNmzdHeno6zMzM8OrVK2RlZWH//v3o1asXAPUM2UlPTy8Tkb+lMjIywoQJE/K94TZ37lz8/PPPCje0vv32W2zfvh1Xr16Vm1KmOMhkMvj5+anUOCx0+uJ5+Mk0xcDAgCZMmKCwfMKECSSTydQWxUqZuLg4yYNvHz16RDY2NjR8+HB6+fJlkdJeu3aNypcvT4IgUMeOHWnJkiUUEBBAS5cupU6dOokROa9fvy6pbHm5fPkyGRgYUIUKFcjT05MEQSAnJyfq3LkzWVlZkSAI5OHhQT4+PgppU1NTyd/fXy7QgCAIVKNGDVqwYEGeQRqkSk1Npf79+4tdn7p160bv378X11+7do0EQaAFCxaoLU97e/tCvapXr662PImIQkJCCpwE+MGDBxQSEiK37M6dO2RhYUHLly9Xa3kKcv78eY0EwmHFR+oxdXBwkIuGqyyo15gxY8jKyqpEPkdx2rJli1hn3717V27dnTt3qGPHjiSTyWjLli1qyzMgIIAEQaDatWvTrl276PHjx5SVlUVPnjyh3bt3U506dUgmk9G2bdsU0kqtV1TBdUPxunv3LllYWJCWlhY1bNhQ7LZXp04d0tXVFX/T3d3dNV1UIvoY5O3JkyeUlpYmLgsPD6euXbtSnTp1qFOnTnT06FG5NJ/TkB2p3NzcyMvLK99tevbsSR4eHgrLU1NTqVu3btS4cWM6evQoPX36tLiKSampqSoNySlKem4cfmLy6lOs7CJD3WQyGfn6+kpK6+HhQQ0bNiSZTEb6+vpUt25dcnd3L9Q0Av379ycdHR06fvy40n0fP36cdHR05KJ8qYO6wkqnpKTQw4cPKSUlRa3lUyY5OVlpPs+fP6fo6GiFSLe+vr60atUqSXnZ2tqSnZ2dwsvU1FS8qKlcuTLZ2dlJ2n9eZDJZgf3qFy5cqPB98PX1FaOr1qhRg/r06UO+vr4Kr4IizRZVixYtSEtLi3744QeKj4+XVPm/fPmSHjx4oNZyMemkHlM9PT2aMWOG+F5ZvT1jxoxPJnR83759SRAE0tHRoVq1apGHhwfVqlWLdHR0SBAEGjhwoFrzc3FxoWrVquUZ0TspKYmqVatGLi4uCuuk1iuq4LqheA0ePJh0dHTEqWFy/oYnJCRQz549ycHBocDpDEqzihUrysUbkOr169d08eJFCg0NVUOpSpewsDDS09PLc1qtX3/9lfT19Sk8PFxhXe7xxnm9co4dLwt4zCFTG1NTU8lj1oKDg8X/p6Wl4ebNm2I01ZyUdTkNDg5G37590blzZ6X77ty5M/r27av2LnlhYWHw9PRE3bp1xWX0/7snGRgYYO3atYiIiMCcOXOwa9cucZs3b97A0NBQfF+hQgVUqFBBrWXLi5GRkdLlFhYWsLCwUFi+a9cuTJ48WVJesbGx+a6bMmUKnj59ir///lvS/vNChegpn5WVpXAu5exufe/ePdy7d09pWkEQsHnzZpXKmNPly5cxcOBATJ8+vUjpkpOTMW/ePOzZswcvXryQ6/p07tw5sUtikyZN1FZWVjhSj2mFChXw7NmzfLe5e/eu2qPgacrevXuxbt06rF+/Hjdv3sTt27cBfBxb9c0332DMmDFqze/69esYPnx4nt2/jI2N0adPH2zatElhndR6RRVcNxSvwMBAdO3aFW5ubuKy7ONcqVIl/PHHH2jQoAHmzJmDX3/9VVPFVEmHDh0QHBwsN9SmKGJjYzFx4kT89ddf4vmdfS6Fh4djxIgRWL9+vVx05rLm9OnT8PDwwKhRo7By5Uq5aKXh4eG4desWOnXqhMDAQLlxh4IgwNXVtVjGRWsaNw4/QWFhYfjhhx8UlgHA8uXLlf7I5R64LoWrqyvOnTsnKW1WVpbkfJOTkwscEG1vb4/k5GTJeeSVr5Sw0tbW1vDy8oK3tzc6duxYpEHCqrp8+TJ2796Nmzdv4t27d2JFFxcXh3PnzqF9+/YwMzMTt69RowYeP36s9nLY2dnhjz/+gKOjI+bOnYsff/xR7Xnk5/bt2woXiPfv3y/RMmQzMzMr8sV+YmIiWrZsiVu3bqFx48awtLTEjRs3xPUNGzZEeHg4du7c+dlfAGqClGMKAM2bN8eRI0eQlJQEExMThfXx8fH466+/xPFEZZ0gCBg3bhzGjRuHt2/fIjk5GcbGxhqd4kOVCz1l9YoquG4oXi9evJAL3qKtrS03hZSenh46dOiAgwcPaqB0H8cLSpU9ZdjSpUvRokULjBw5EsuWLZP7fS/IgwcP0Lx5c7x8+RI9e/bEkydPEBkZKa53cXHBixcvsHv37jLdOPTz8xP//99//8kF/cl24sQJnDhxQm6ZIAhy48FL2p9//ok//vgDFy5cwIsXLwB8nELD2dkZgwYNgpeXl+R9c+PwE5T77kZOM2fOVFgmCIJaGodLlixB8+bNsWDBAsyZMwfa2iVzelWuXBlRUVH5bnPu3Dmlcz6pwsrKSpy7DwAqVqwo3vnOlpqaqjBfYY0aNbB7927s2bMHlpaW+PLLL/H1118X+w/1jBkzsHLlSrngC9mICIMGDcLKlSsxceJEcfnQoUOxdOlSPHr0CFWqVFFreXR0dNChQwfs3btX5cbh0KFD5d4fPHhQ6ZPLzMxMxMfHIzQ0FF26dJFblzNya0ny8vLCmTNnkJWVVegbBX5+frh16xb27NmD/v37w9/fXy64kIGBAdzc3HDmzJniKjbLh5RjCgDTp0+Hh4cH2rVrhzVr1oh36N+9e4fIyEiMHz8eGRkZmDJlSnEVXWPKly9f7I3CevXqYf/+/fj+++/lem9ke/36Nfbv34969eoBUE+9ogquG4qXhYWFXDRgCwsLheOrra1dotEoc3J3d5d8syK70fL111/DxMQEW7ZswY4dO2Bvbw9ra2uF/SoLajR//ny8evUKISEhaNmyJfz9/eUah9nzaKszYJQmlLUgfElJSejduzdCQkIUHvbExcUhLi4O+/fvR9u2bbF///48e4zlS2MdWlmxCAgIkPRSB19fX2rdujXJZDKqXLkydenShXx8fIp9zNbEiRNJJpPRt99+KxdkhYjo/fv3NG/ePJLJZDRp0iS15tupUye5AcpffvklGRgYUEREBBERxcTEkKmpqdI5Da9cuUJTp06lypUri33V69atS4sXLy4w6IEU2cEfPD096erVqzRnzhyFsTEtWrRQGNN5//596tatG9nY2NDatWvp3LlzFBsbS3FxcQovKYYPH66WuRVzB2coKHiDi4sL3b59O999ltRYneTkZGrSpAkNHDiQHj58WKg0tra21KNHD/G9srFp48ePJ3Nzc7WWlRWOlGOabf369aSjo6N03IqOjk6e42JYwbLrwfr169Off/5Jz58/J6KPY6737dtH9evXJ5lMJv4mFke9UhRcNxQvNzc36tKli/i+Z8+eZGxsLAZIevbsGVWuXJkaNGigkfLNnz+f/Pz85F6dO3cmQRCoVq1a5OvrS7NmzSJfX1+qXbs2CYJAnTt3Jj8/P3Ef+Z2zBQU1qlSpklysBmXn0uTJk8nExKT4/ghlRFpaGh07doxWrlwpF9jv/fv39PTpU8rMzFRbXtkBEL/44gvasmUL3bt3j1JTUyk1NZXu3btHmzdvpi+++IIEQaDevXtLyoMbh0xtVKmEssXHx9P27dtp6dKl5O/vr/BSFk3zxYsXVKNGDZLJZGRpaUndunWjoUOHUrdu3cSooTVq1ChyFNSCrFmzhrS0tCghIYGIiKKjo0lfX59kMpkYAU0QBDpw4ECe+8jKyqJTp06Rt7c3VahQgQRBIC0tLXJzc6NNmzapraxNmzalL774gj58+EBEyit5X19fqlatmtyynBdF6h5sHRoaSuXLlydHR0fJnytbbGwsxcbG0v3790kQBJo8ebK4LOcrPj6e3rx5k+d+kpKSaMKECWRlZaXwuaKioqhLly70zz//qFze3C5evEimpqYkk8nI3Ny8wKiuurq6NHPmTPG9suM5bdo0tTS8mTRFPaY5xcTE0IQJE8jFxYUcHByoSZMmNHbsWLp27VoJfwr1yf682RNmayqa8fjx4+XqM21tbbmgEjmjfaurXlEF1w3FZ+nSpaSvr0+vXr0iIqKgoCCSyWRUvnx5atq0KZmYmJBMJqPffvtNswX9/0JDQ8XAKcoma88OnHL27Fm15Kenp1fguTRp0iQyMDBQS35l1aFDh8ja2lruWinbuXPnSCaT0c6dO9WSV1hYGAmCQB06dFB4GJLT+/fvqV27diSTycQHFkXB3Uo/c+np6Thw4AAGDhyo8r5UHbM1ffp0rF69Wq4PN+UYRJ39/9yTw5ubmyMqKgozZszAnj178Ndff4nr9PX14evrW+S+9oUxevRo9O/fH6ampgAAR0dHnD59GosWLcK9e/fQpEkTjB8/Ht26dctzH4IgoEOHDujQoQPev3+P//3vf/j9998RGBiIsLAwDBs2TC1ljYmJwYgRI/Lt6mttba0QDGPw4MGSu7W0bdtW6fKMjAw8evRI7L4jdc6enHJ2Cd26dSucnJyK3E1UU2N1Tp8+jR49eiA1NRU6OjowMDBQOi445zJzc3PEx8fnu9+bN2+KE3izkiXlmOZUt25drF69uriLWaJyB2spbPCWvP5GUq1Zswb9+vVDQECAwhyAQ4YMgaurq7itOuoVVXDdULzGjBkDd3d3aGlpAfjYjXPPnj3w8/PDtWvXYGtri4ULF2LEiBEaLulH3333Hbp164bhw4crrBMEASNHjsSJEyfw3XffqaWrpLW1tcJQmdyuXr0KGxsblfMqDZ4/f46YmJg85xMFPl4T5RQeHo6+ffuiUqVKWL16NaKiouTiTDRr1gw1a9bE/v37MWjQIJXLuHv3bmhra2Pz5s3Q19fPczt9fX1s2bJFHMZU5Hm+JTRc2ScgOjqaxo8fT2ZmZqVijqTffvtNnPPqwIEDJAgC+fr60p49e2js2LGkq6tL/fv3F0NO5yU9PZ3+/fdfCgsLo3///ZfS09NL6BOoLiUlhbZs2UIeHh7inWx1MTIykpvrSNkdwMGDB6u1q1F+T47Nzc2pc+fOdOrUKbXlp6rsJwp//PEHESn/G/Xo0UMtTzpzatKkCenr69PevXsV7gbnZcCAAVS+fHmKj49XWtbr16+TtrY2jRgxQq1lZYUj5ZgylhvXDSwnQ0NDmjt3br7bzJkzhwwNDdWS37Bhw0hXV5euXLlCRIrnUmhoqPhEvSx79+6d+Fnz6h2VV6+3Tp06kaWlpdhFXdl1w6BBg6hGjRpqKWuzZs3Izc2t0Nu7ubkpnZqnIPzk8DOSnJyMXbt2YfPmzbh8+TKICAYGBmq5m6FMYmIi3r59i2rVqhW47W+//QY7OzscP35cHHhvZ2eHAQMGYMCAAejfvz86dOiAfv365bsfHR0dNGjQQC3lLwmZmZn466+/sGPHDhw9ehSpqamQyWTo2LEjvL291ZZPgwYNcObMGWRmZop3SXPKjlyqzidiqkSgVcXKlSuxZMkS/Pvvv0qDECUkJMDR0RHfffcdJkyYIC4/fPgwunfvjv79++e5bzs7O0RERKi1vDExMfj6668LPLdzmjt3Lg4dOoRWrVph8eLFYqSyGzduICIiAnPnzoWenl6RQ+Az9ZByTHN68uQJLl68iKSkpDyj4eW+g82Kl9R6RRVcNxSv7du3w8nJCQ0bNsxzm2vXruHSpUul4vumq6uLy5cv57vN5cuXoaurq7D84cOHCAoKQkJCAtLS0hTWK+uV9e233+LPP/9EmzZtMH36dNy5cwcAcPz4cURERGDVqlWwsLAo8+fShAkTsGXLFjRs2FB8CljYgIrnzp1D3759lU4Flq1atWo4dOiQWsp6//59DBgwoNDbN2zYEH/88UfRMypyc5KVOWfOnKGvvvqKypUrJ94FqV69Ov3222+UnJys1rykjtkyNDSUe7IlCAJ99913ctt4eXlJugOiLsqCsBT2lVtkZCR98803ZGlpKd6VatSoEa1atYqePHmi9rJv3ryZBEGg4cOHU2pqqtzdreTkZOrXrx/JZDLat2+f2vMuaS4uLgqBdXJr3749tWjRQm6Zpsbq2NjY0Pjx44uc7tChQ2RkZKRwZ1MQBDIyMqJjx46ptZys8KQe0/fv39PXX38tNw6usHewPxXJycl06tQpCg0NVfmpa3b9mz15fFHq7CdPnsgFkZBar6iC64bilXPS+7wsXLiw1HzfBg4cSDKZjJYsWUJpaWly69LS0mjx4sUkk8noyy+/lFs3bdo0hSBXOcfd5lenREVFkZ2dnUL8AUEQyNbWli5cuFBsn7ekmJqakrOzs1hPFIWBgQFNmTJFfK/sumHUqFFUoUIFlctJ9HEcaO5r4/x89913kq5Z+MnhJ+rRo0cICAjA1q1bcf/+fRARqlWrhq+++gpLly5Fu3bt1N6PXtUxWznn9Spfvjxevnwpt7527dp5TtERFhaGFStW4MqVK0hISBBDwOeUc/JWKezs7CSNv8udr4ODA+7duwciQpUqVTB9+nR4e3uL4dOLw9ChQxEYGIjNmzfjjz/+EP/WzZo1w40bN/D27Vv4+Pigb9++Cmlfv36NtWvXIjAwMN+7jnfv3s0z/5cvX+LKlSviPGaOjo4wNzdX2+fL6fbt2/jqq6/y3aZevXrYuXOn3DJNjdX56quvsG/fPrx//x4GBgaFTufp6Yn79+9j27ZtOHfuHBITE2FkZAQXFxf4+vrmeyeTFS+px3TWrFnYuXMnatWqhS+//BJVq1YtsSmBStrGjRuxY8cOHDx4UBy3feXKFXTp0gVPnz4FALRo0QKnTp1CuXLlJOWRXWffuHEDtWrVKnIdrquri549e+LXX3+VXK+ogusGzcvMzCzRuYjzs3z5cpw9exZz587F6tWr0bRpU1hZWeHZs2f4559/8OzZM1SuXFlunuuNGzdi5cqV6NChA0aPHo0+ffrAx8cHnTp1QmhoKDZt2gQvLy+MHTtWaZ4uLi64ffs2jhw5onAu9ezZU+lTyrImMzNTbuxpUVSvXh0XLlzId5vIyEi5+TRVkZ6eXqTfBC0tLaSnpxc5n0/zV+cztn//fmzZsgWnTp1CZmYmDA0N4e3tjcGDB8PDwwOCIGDp0qXFkrcq8ytVqVIFDx8+FN/XqFED586dk9vm2rVrSufB+v333+Hj4wMiQvXq1dGsWbNiuaBSJThLTk+ePMHgwYPh7e0tHpOSsGvXLnh4eGDt2rW4du0aiAj//PMP6tatiwkTJmDUqFEKaZ4/f46WLVvi7t27MDIyQkpKCoyNjZGeno73798D+DjPpI6OjtI8Y2NjMXHiRBw7dkwuaIIgCOjevTt++ukn2NnZqfVzvn//vsD50vT19fHmzRu5ZW3atMGhQ4fw8OFDVK1aVSFNTEwMTpw4AV9fX7WW18/PDzdu3EDnzp2xePFiODo6Kp2DTRkzMzNMnjxZreVhqpN6TPfu3YsvvvgCFy9ehJ6eXgmUVHN+//13pKeniw1DAJg6dSqePXsGX19fPH36FH/99Rd++eUXTJ06VVIe2XV29sT0RanDU1NTcfv2bezduxflypWTXK+ogusGzbt8+bLag9lJVbVqVfzzzz+YNWsW9u7di2PHjonr9PX14e3tjaVLl6JixYriclWG7CxYsAD29vbw9vZGr1690KtXr+L/kBrg7OxcYOCdvPTp0wcLFy7E1q1blV4brFixAteuXZNrsKuqRK4Zi/yskZVq2VMhdOrUiXbs2EHv3r1Tuk1xDEZXZX6loUOHkoODg/g+e27CkSNH0tGjR2nWrFkkk8moX79+CmkdHBzI3Nyczp07p8ZPU3yUHRNNlOHRo0f0+vXrfLcbO3YsCYJAv//+O2VlZcl1wzl//jy5uLhQq1at6O3btwpp79y5Q9bW1vnOyWRtbS3OKaUutWvXpjZt2uS7TZs2bahmzZpyy/7991/S19cnGxsb2rFjB40bN45kMhnFxMTQpk2byNramsqXL0+3bt1Sa3mVdfOROl1IVlYW3bp1q0TmZ5QiJSWFFi9eTG3btqU6deqUyNQFmiD1mBoYGNDUqVM1VOqSZW1tTSNHjhTfv3jxgmQyGY0aNUpc5uLiQo0aNdJE8UR9+/alypUrS65XVMF1g/rrBg8PD/ElCALZ29vLLct+tWnThqpXr04ymYwGDhyohk+oXunp6XT16lUKCwujq1evKnQzzabKkB0dHR2aPn26egteCkVERFC5cuXoyJEjRU77+vVrqlevHslkMmrfvj25urqSTCaj6dOni/N+N27cmFJTU9VSVkEQSEdHhwwMDAr1yu5OXFT85PATRER4+vQpnj9/jtevXxepO4oqHj9+XOCUGHp6enj79q3Ccm9vbyQkJCAuLg62traYPn06jh49io0bN2LTpk0gItjZ2WH58uUKaePj4zFs2DA0a9ZMbZ+lOJXU8SioDIUpx19//YV27drh66+/Vljn7OyM48ePo0GDBvD398eyZcvk1s+cORPPnz/Hhg0bMGLECLm7XUSE3377DWPHjsXMmTOxb98+1T/U/9etWzf89NNP2LJlC4YOHaqwftOmTQgLC8PEiRPlljdo0AB//PGH+KQ9u5z169cHEaFChQrYu3cvHBwc1FZWAHB1dS3yncADBw7g4MGDWL16tfjkJTY2Fj169EBMTAwAoF+/fti5c6ekrjLFQdWn0GWJlGMKfOw6n92l8lOXlJQES0tL8f3Zs2cBAL179xaXtW7dGlu2bCnxsuXUsWNH3L17Fx4eHpLqFVVw3aD+uiE4OFj8vyAIiI2NFadVykkmk8HMzAz9+vXDTz/9pHK+6qajo4P69esXalupQ3ZsbGyQlJSkSjHLhOzu656enmjcuDEcHR1hZGSksJ2yoD2GhoY4e/Ysxo0bh71794oBxFasWAFBENC/f3+sX79ebT1BbGxs+MkhK7rQ0FAaPHgwlS9fXrzD0LVrV9qzZ49456K4nhxWqlSJBg0aJL5X9uSwe/fuZG9vX6j9paen0759+2jp0qW0e/fuPCcZrl27tsbDcl+6dIk2b95Mq1atos2bN9OlS5fEddu2baNt27ZRSkqK3PvCvNQtISGBVq1aRd7e3uTp6Une3t60atUqSkhIULq9np6e3J1DbW1tmjVrltw2w4cPJzs7O4W0JiYm5OXllW95PD09ycTERMInyduzZ8+oUqVKJJPJyMPDgxYvXkzbtm2jxYsXk7u7O8lkMqpatSo9e/ZMafqXL1/SqlWraMCAAdShQwfq06cP/fDDD2Ko6tKgffv21KBBA7llXl5eJAgCtWvXjpycnErV5M1Eqj2F/lzs3LmTKlSoQLdv3y7xvB8/fkzr1q2j8ePH07Bhw8Tlz549o3Pnzqm9x0PlypVp6NCh4vtx48aRjo6OXD0/ZcoUtYXlV5Wq9UpJ4bqh8AoTkCY/Jf2dkZJv7dq1ydvbW3zv6OhITZo0kdtft27dyNraWiGfuXPnUrVq1SgpKakYPkXp8eLFC3J1dc1z+q2c03AVtJ/jx4/Tzp076ciRI8USXLCkcOPwE5WSkkIbNmwgZ2dn8aQ2NjamYcOGFVvjUFPzK61YsYIqVapEL1++VOt+C+P8+fPk6OiotPuPo6MjXbhwQVz233//EREV2EWouCISrl27lgwMDMT953wZGBjQunXrFNJYWVnRpEmTxPeWlpb09ddfy20zZcoUMjAwUEhboUKFQs3JpK4oXjndvHmTmjZtKlepZ/+/WbNmGrn4VqfKlSvLXRSkpKSQjo6O2P0pPT2d6tWrR82bN893PxEREbRgwQIaPXo0+fr6KrxyXryrys7Ojtq3by++z31hlpiYSFWqVKEZM2aoLc+yJiQkhPr27Uumpqbk7+9PR44coZCQEKUvdVq3bh3p6+srvQi6du1asTQmOnfuTObm5nT16lW6ffs2mZmZkbu7u9w2vXv3lhtuoA7Xr1+nMWPGUNOmTalmzZpF6r5YFuoVrhsKLzg4mGJjYyWl1cR3Rkq+qgzZSU1NpW7dulHjxo3p6NGj9PTpU7V/ntLgyy+/JEEQqFu3bvT7779TYGAgBQcHK319Lrhx+Bm4evUqTZw4kSwsLMQKpVKlSuTv7y+5YlRGU2O2iD7eeaxTpw7t2LGDrl69WugpJVRx8eJF8Qlt+/btyd/fnzZs2ED+/v7Url07EgSBDA0Nyd/fnwICAsQnhwEBAYV+qcvu3btJEASytLSkhQsXUkhICN28eZNCQkLo+++/JwsLC5LJZOIE8NmaN28u9/SvY8eOZG1tTS9evCCij2MXHRwcqHbt2gp5urm5FfjksGfPnuTh4aGGT6jc+fPnae3atbRo0SJat25dqQ27nTNkfraIiAiaM2cOLViwQLzhkk1fX1+u4f3XX3+RIAh06NAhcdmkSZPI0tJSaX4fPnygvn37KoQmz33Rq84bFFKfQmd/j/73v//luW8/P79CjbsqSUU9pkSkcCzyu4GkLocPHyZBEMjZ2ZmOHDlCY8eOVdi/k5MTde3aVW15En2cYin3Zzp8+LC4PjMzk6ysrNQ63is4OFi8sNbR0aGqVauSnZ2d0ld+SrJe4brho5KsGwqaTkVT3xkp+QYFBVHnzp3Fa73Xr19T48aN5Y6lvb290mtBdY55Lc1MTU3Vch2SXw+ysobHHH4G6tevj59++gk//PAD/ve//2HLli04ffq0GE3Uzc0Np0+fVjkfVcdsPX/+HFu3bsWFCxfynPxZEASlZW3cuDF27dqV70S1qk5lkdvs2bPx4cMHnDx5Eh06dFBYf/LkSfTo0QMRERGYN2+euHzIkCFqK0Nh/fDDD7CwsEB0dLTcBM61a9dGmzZt4OPjg0aNGmHZsmVyk8B37NgRP/74I969e4dy5cph1KhR6Nu3LxwdHdGiRQtcunQJsbGxWLRokUKeixYtQrt27bBp0yYMHz5cYf1vv/2GkydPquXcy4uzszOcnZ2VrgsNDQXwcToPfX198X1htGnTRi3lA4DJkyfjl19+wZMnT8SxIX/++ScGDhyIrKwsAMDPP/+MS5cuiVFUjYyM5MaNBAUFQSaTwdXVVVymo6OjdHwv8HFC7/3792Po0KEYO3YsmjZtikmTJmHAgAEIDQ3F0qVL0b59e4VxpKowNjbGhw8fxPempqZyEYqzP5eyMXdv375F3759sWTJkjwnXKYc0XA1TcoxBYB58+aVWPTibMuXL4eNjQ2CgoJQvnx5XLx4UWGbBg0aiGMC1cXDwwOHDx/G1q1bAQADBw5Ejx49xPXh4eGoXLmy3BhEVc2aNQsZGRnYtGkThgwZInnMXX71ijpx3fB/iqNukDqdiqa+M1LydXd3h7u7u/je0NAQUVFROHToEO7evQtbW1v06NFDaSReqWOnyxoiQtOmTSWnv3DhAkaMGIGrV6+K+8v+uzVo0ACbNm1Saf85bd++XVK6/K6NldJky5Spn729Pa1evbrA7eLi4mj+/PlkY2Oj9u6LUsZsXblyhczNzZV2eSyoz/eaNWtIJpORnp4etWvXjr7++mvy8fFR+lInQ0ND+uqrr/Ld5ssvv1ToNhkSElLgU8wHDx6oteuYvr4+ffPNN/luM2bMGIXuoQkJCbRnzx65Y7dixQoyMTEhQRCoXLlyNG3aNKWTx/r7+1Pnzp1JJpNRnTp1aNiwYTRnzhwaNmwY1alTh2QyGXXp0oX8/f3lXgsWLFDPhy6AlO6+6n5qQ/RxDEiXLl3kltWtW5fMzMzo999/p+XLl5O2tjZNnDhRXN+mTRuqUqUKvXjxgl69ekVVqlShZs2aye2jf//+eY7vbdCggdy4pNzduK5du0blypWjDRs2qOETfiT1KbQgCDRq1Chq0qQJyWQyGjZsmML5pmx8syZJOaaaUqFCBRo7dqz4XtnfctasWUq7jpc1BgYGCt3iSzOuG4q3bnB1dVWI0tmuXTvS0tKiYcOGUffu3Ukmk9GKFSvkttHUd+Zz+q6WpG7dulHHjh0lpS1sD7LLly+rpayFvU7JuZ2UJ7v85PATExsbW6joUjY2NvDz88P8+fPznFheKinzK02dOhWJiYn49ttvMWzYMFStWrXQd3V//PFHVKlSBREREUrnpysu2trasLW1zXcbe3t7hc/h4eGB+fPnyz1NzG379u2YN2+e0qenUpiYmBQ4R5ehoaFcVDMAqFSpEgYMGCC3bOrUqZg0aRJevHgBKyurPO8s+vn5if//77//8N9//ylsc+LECZw4cUJumbKIYFJERkYiMDAQCQkJSEtLU1jv6OgILy8vcUJoTTy1AT5G23VzcxPf379/Hzdv3sT8+fPFKLFnz56V+ztNmDAB/fr1Q9WqVcWnAAsXLpTbb1RUFBo3bqw0zzt37sg9zRUEQe7Ofb169dCjRw/88ssvSue/lEKVp9CVK1fG2bNn8dVXX2HLli2IjY3Fn3/+qXC+lhZSjqmmZGVlFRgF8tmzZ5/EvIvly5eHlZWVSvsoqF4RBAGbN29WKY9sXDcUb91w69Yt9OzZU3z/8uVLBAUFYcSIEdiwYQMAoHnz5ti5c6fcXJua+s5IyTc5ORlxcXGoWbOm3NPPbG/fvsXdu3dhZ2enEKGzbdu2aN26tdx81Z+iFStWoGXLlli7di3GjRtXpLSF7UE2a9YstdT3ixcvLtR1SkZGBgICAnD37l1J+XDj8DMnCILSE7qkRUZGwsvLS1Il9OTJE4waNapEG4YA0LJlS5w7dy7fbaKiotC6dWu5ZVSI7m9ZWVlqbah4enriyJEjWLRoEbS1Fb/2Hz58wJEjR+R+KIGPXS/t7OxgY2Mjt1xLSwvW1tYAPl7A3L9/X6GrZVBQkNrKXxQZGRn48ssvceDAAbF7R86/efZ7QRBw+fJlcXnOxmxJevv2rVzDPSQkBIIgoEuXLuKyL774Qq77bZ8+fbBu3TrxInTgwIHw8fGR20dKSgo6d+6sNE9dXV25CwVDQ0M8e/ZMbhtbW1scOXJEpc+W0+jRo/HFF1+IF4C9e/fG8uXLsXDhQuzfvx8GBgaYMmVKnl3DDAwMsH//fsyYMQMrV65Ey5YtcfToUVSvXl1tZVQXKcdUU2rXrp1v97eMjAyEhoaiQYMGas23KN2jitwlKg9du3aV3NWvKPWKuhqHXDcUb90gdToVTX1npOS7YMEC/Pbbb0hISFCaJjMzE61atcLYsWMVugqfO3cOzZs3V0/hS7EffvgBDRs2xMSJE7FmzRo0bNgwz6kscn+3IyIi0K9fvzyvozt16oS+ffvi6NGjainrrFmzCtzmzz//xLfffou7d+/C2NgYM2fOLHpGannOyUoNVUMzF0V21Lz379/LvS/MKzdTU1OaNm2apHI0aNBA7V1GC+P69etkYmJCc+fOVQix/fbtW5o9ezaZmppSTEyM3LrCHCNvb28yMzNTW1mTkpKoadOm5OHhQZGRkXLrIiIiyN3dnZydnRVCVstksgLLunDhwlLVpW/p0qUkCAINGzaMLl68SIIg0OTJkykqKop++OEHMjMzo/79+9P9+/c1XVQi+tgVvHfv3uL7AQMGkJGRkVz3qLFjx6r1fKhfv77ctDPNmjWjOnXqyAVfaNOmDVWtWlVteeYlIyODnjx5ojTwA5Hy78vGjRtJR0eHLC0tKTQ0tNR1K1XlmD548IBGjhxJ1atXJ319/WIPALFy5UoSBIH8/PyISL6rWkZGBk2cOJFkMhlt3LhRbXkSaSZq89OnT8nBwYHGjx9f5GkRNFGvcN1QvHWD1OlUNPWdkZLvF198Qb169cp3v3369KH69esrLG/cuLHcufCpKmgKi/yGNZmYmNCcOXPy3f+cOXPUPmWXMqGhodS8eXOSyWSkr69PU6ZMkRzFn58cfoJKqmucu7s7BEHAjRs3UKtWLfF9YeTuLunm5oZ//vlHUjkmT56MqVOnIi4ursBunqpQNvFxw4YNsWTJEvzyyy9o1KgRrK2t8fTpU1y+fBlJSUlwdXXFihUrFJ4WHjx4UOnEu5mZmYiPj0doaKjc3eGiUnbHND09HZcuXUKrVq2gra0NCwsLvHjxQgzSU6lSJTRu3FiuG0Lucisj5SlnWloaZDJZsUx4vnPnTtSvXx+bNm0Sl5mYmMDFxQUuLi7o2rUrmjVrhrZt28p1i4qJiUFgYCC+/PJLubvJ2Z49e4Y9e/agQ4cOqFu3rtrK6+bmht27d2Pt2rXQ19fHgQMH4OXlJdcd+e7du2p9Mu7m5oZDhw6JTzoGDBiAadOmoXv37ujatSvCwsIQFham9JyXaujQoWjQoIFCl/OcT6ELa/jw4bC3t0ffvn3RsWNHNGzYUG3lVAepx/TevXtwcXHBq1evUK9ePaSlpcHW1hb6+vq4d+8ePnz4AEdHR7V2px0/fjyOHDmCBQsWYOfOndDX1wcA9O/fH//88w9iY2PRsWNHDBs2TG15AhAD0eSWnJyMS5cuYdeuXfD09JQLUqOqgQMHwtDQEOvWrUNAQABq1aqV5xOC3E91pdYrquC6oXjrhoYNG+LQoUOYPHky9PX1sWvXLrRq1UruaW1sbCwqVaokl05T3xkp+T548ADdu3fPd781atTA33//rTS/cePGISYmBl988YVaP0tpcv/+fclppfYgU6dr165h1qxZOH78OARBwNdff43vv/9eocdXkaix0cpKAUEQyNTUVOncTXm98prTqSDz588nPz8/8c5E9vvCvHK7c+cOWVhY0PLly4tcjpCQEPLy8iILCwtauHBhsc0NVti7S8ruNuX3Xtn2Li4uKs2ZZWtrm2eI9oJeuT+z1KecISEh9N1339GrV6/EZS9evKDOnTuTtrY2GRgY0MyZMyV/xrwYGBjQ+PHjxfcymYy+/fZbuW0GDBhAjo6Ocsu8vb2pcuXKSkPHE328M1u1alW1P6W+ffs2mZiYiOeFoaEhXb9+XVyfkpJCBgYGNGbMGKXpHzx4QFFRUUU65y9evEijR48Ww+Cnp6dTjx495M5DFxcXMSCEOujp6SmEpy+M/M7BmJgYql69erHMC6oKqcd08ODBpKOjI86nlfOzJyQkUM+ePcnBwUGtx4WIKC0tjebMmUNmZmZy54CxsTHNmjWL0tLS1JpfYURERJC+vj4FBgaqbZ+qPCGQWq+oguuG/KlaN6gynYqmvjNFzbdChQoFBr6aOHEilS9fXmF5SEgIdevWjSpUqEDTpk2jP/74g4KDg4t93tWyRGoPMnWIj48nHx8f0tbWJkEQqEuXLnTlyhW17FsgKkXxv5nKZDKZpHTZYbE1ZejQobh//z5CQ0Nhb28PJyenQvf5lslkcuM/8nuKpUqAl7i4OMlpsxERqlevjkmTJmHixIkK67W0tGBqalpg8JjilPOOcEBAAJycnODk5KSwXe6nnLnHoHh6eiImJgZ37twRl/n4+GD79u2oWbMm3rx5g6dPn2L37t1y02eoysTEBKNHj8bSpUsBfAyTPnDgQPz666/iNjNnzsTatWvlwrnb29vD1dU137FQPj4+OHv2rORB3nl5/Pgx9u/fDwDo0aOH3BPwS5cu4ffff8egQYPkwucfOXIE06dPx+3bt/Pdd1HO+X/++UcMb96sWTPJ9Yky9erVg7OzMwICAoqUbtu2bXBycoKjo6PS9S9evMC8efPw/v37PJ9GaYKUY1qlShU4Ozvj4MGDAD7WbfPnz8f8+fMBfHzi3qBBA3h4eMidz+pCRPjvv/+QmJgIIyMj1K1bV/J0D+owYMAAPHz4EOHh4RorQzap9YqquG7ImzrqhmPHjslNp9K3b19x3dmzZzFhwgTMmTMH/fr1U5peU9+ZwubbrFkzpKSk4MaNG0qvjbKyslC3bl0YGhoqTI1RUtdWpU1iYiLevn2LatWqFbjt0KFDcffuXYSFhcHExCTPHmS5e3OpMjY5KSkJixYtwrp165CamgpnZ2csW7ZMbsoSVXHj8BMjk8ng5+eXbyTM0qiwPzSCIChUQn5+foXu1ph9kaVJ27ZtQ6NGjUpdV7hsOY9F7qALuQmCAGdnZ+zYsQM1a9aUW1etWjW0a9dO/MFPTU2FmZkZ2rRpgxMnTuD169do2LAh7Ozs1Bq8pkGDBmjYsCF27twJAHBxcUFKSgpiYmLE88TNzQ337t1DfHy8mC476IGyiHjZ5s6dK0bV06Tg4GC0b98eFStWRJ8+ffDzzz/Dzc0NderUQVhYGK5fv47u3bujSZMmpeKcBz7On7Z06VJER0ejSpUqmi5OqaSnp4fJkyeLDRBdXV1MnjxZLlDEN998g4MHD+LRo0eaKmaJmTFjBtavX483b95ouiiS65WSxnUDy+nHH3/E1KlTMWTIEPz0008wNjYW1yUnJ2PixIn4/fffsWLFCoVuvWXt2koVycnJmDdvHvbs2YMXL17IzYt97tw5+Pv74/vvv0eTJk3k0km9SaLsWrYwli1bhmXLliE5ORk1atTA4sWL5W5oqAuPOWSlgip9vjUVZVKqIUOGaDT/mJgY3Lx5E2/fvoW3t7fC+uxjoepTzpcvX8r90EdGRiI1NRW+vr4AgAoVKqB79+7iXXF1kTpmpnz58gpR+XJ79uyZOM5Dk5YuXSre6bW2tsbPP/8MDw8P8abQkiVLsHDhwlIVgrxPnz4ICgpCy5YtMWPGDDg7O8Pa2lrpxYeysRLp6ekIDAwUz93s6U5SU1ORkpICCwuLfH+oCzrvSwMLCwu5p04WFhYKY5O1tbULNV1RWUdECA0NhYGBQZ7blOQx1cRYPCm4bih63fApGzduHPbv349t27bh0KFDcHZ2RpUqVfDo0SNcuHABSUlJaNOmjdIpHMratZVUiYmJaNmyJW7duoXGjRvD0tISN27cENc3bNgQ4eHh2Llzp0LjUJVrVylmz54NQRBQo0YN+Pj44N69e/jhhx8KTDdjxowi5cNPDj8xmnxyWNiQ8oIgqL1bnipiYmKwdu1asaJUdjcnvzKnpqbiwoULec57BSgPxS41nVQXLlzAiBEjcPXqVXFZ9mcNDQ1F586dsWfPHnh6eorrC+q2kx9zc3N89dVXWLNmDYCP8wguWrQIjx49QsWKFQF8rOh++uknvH//XpWPJufSpUvYuHEj5s6di6pVq+LDhw/o06ePXCjpZs2a4dixYzA3NxeXdezYEZcvX8bt27eVBvx49eoVHBwc4OTkpNLcoAsWLIAgCPjmm29gZmZW6Is0Icf8j+bm5ujRo4f4VFYmk2HevHlyP+atW7eGmZkZDh8+rHR/qp73RZWzi1J+d6Nz3rHNdvjwYYwcORLPnz8X02eX9/z582jRooXYvS43Ked9UanjmAIfg3yVK1cOf/31FwDAy8sLwcHBuHTpEqpXr47nz5/DyckJ5ubm+Pfff+X2lZ6ejoMHDxZ4PHN3ZVK13pZ6HoWGhirNJyMjA48ePcL27dtx5swZDB48WKFLoKrH9OHDhwgKCsp3rsLcc61KrVeKguuGkq0bshX1t1hT1zpS801NTcW3336LjRs34vXr1+JyIyMjjBo1CgsWLChV85hKrcukmjBhAtauXYs9e/agf//+8Pf3x4IFC+Ty9fT0xIMHDxAdHa2WPKUqSs+unNsV9SklNw4/MZpsHNrZ2Smt2JOTk8U73ZUqVYKurm6x3G15+/YtDh48iOjoaKSkpMDIyAhOTk7w8vLK8+lWSEgIOnfujLS0NGhra8Pa2lrpPICA8jtE69atw3fffYfk5GSlaXL/WKmaTqrr16+jefPmkMlkGDFiBG7evInjx4+L+yci2Nraws3NDb///nuB+yMi3LlzB/r6+nn2y8+Ounj16lUIgoD69etDT09P7oLuq6++QlhYmFrGcxakoDEzhw4dQq9eveDo6IjVq1fLzdsYEhKCiRMn4urVq9i/fz+8vLwklyP7Qig7yq+ULtW5u8AaGBhgzJgxWLVqlbj9tGnTsHXrVrx8+VJhX6qe91L4+PgUuotSzsZAeHg4PDw8UKlSJUyfPh1RUVHYvXu33Hejdu3aqF+/vsJTaHWf93lRxzEFPnYZ8vPzw+PHj2FiYoLg4GC0a9cOBgYGqFu3Lu7cuYOUlBRs2LABI0aMENPFxcWhQ4cOuHv3boHdwHPXKarU26qcR9l/s7wQEVq1aoXDhw/D1NRUXK7qMZ0+fTpWr14t93fI2Sgpat2rzrF4XDcUTF11QzYpv8WautZRNd/MzEzcvHkTycnJMDExQe3atQs1PlLKtZVUqtRlUtnZ2aFhw4bizRJljcMJEyZg165dePHiRaH2WZhrJCm2bdsmKV1Re6xxt1KmNsqmZsi5bsqUKXj69KnSkMnZdu7ciYCAALlKqFGjRvDx8cn3zt/+/fsxcuRIJCUlKUxMbGJigo0bN8pNbJtt1qxZyMjIwKZNmzBkyJAiDSQ/cOAAxo8fjwYNGuC7777D1KlT4eXlBRcXF4SGhuL48ePo06ePQhhpqelUkT0e4OLFi6hZsyb8/f1x/Phxcb0gCGjRogUuXLigUNaDBw9i9erV4gVabGwsevTogZiYGABAv379sHPnToW/3YgRIzBy5EjUrFkTurq6iI2NxfLly+W2uXjxotpDZG/fvh3W1tbo1KmT3PKmTZuiadOmeabr2bMnJk+ejB9//BEeHh7Q09NDxYoV8eTJE6SlpYGIMH36dJUahgDE8ZXZ3aOkjLesWLEinj9/Lr6vUqUKrl+/LrfNy5cv8/zxVOW8l6qowSayff/99zAxMcHFixdhYWGh9IK2adOmSsOJSz3vi0odxxQAxowZA3d3d/F4uLu7Y8+ePfDz88O1a9dga2uLhQsXyjUMgY/T+dy5cwfe3t4YOnQoqlatmucFfW6q1NuqnEfz5s1TeqErk8lgamoKZ2dnuLi4KKxX5Zhu3LgRK1euRIcOHTB69Gj06dMHPj4+6NSpE0JDQ7Fp0yZ4eXlh7NixCmml1itFwXVD0ahSNwDSf4vVca0jhar5amlpoV69ekXKU+q1lVSq1GVSPX78GAMHDsx3Gz09PaWBplS5RpKixIYlqSXmKSs1YmNjFSYyLy3S09Opbt26NGnSJIV1GRkZ1Lt3bzFkt4GBAdnZ2ZGBgYEYirp3795KpxkIDw8nbW1t0tPTo5EjR9Lu3bspODiY9uzZQ6NGjSJ9fX3S0dGhiIgIhbQGBgb09ddfS/o8rq6uZG1tLYYvzh1We+fOnaStrU1BQUFqSacKCwsLGjJkiPhe2cTA06ZNIyMjI7ll7du3pwYNGsgt8/LyIkEQqF27duTk5EQymYx+++03hTyzsrJoxowZZG5uTubm5vTNN9/IHb+IiAgSBEHS9CX50dbWVnqOFdaRI0eoS5cuZGFhIU6m3K1bN/rrr7/UWErV9OzZk1q0aCG+Hz58OOno6NCuXbvozZs3dOLECSpXrhy5uroqTa/KeS9VXFwcJScn57tNSkoKxcXFyS0zMTGh4cOHi++VnbszZ85UGopd6nlf1hgbG1P79u2LZd/51duaOI9UOaZNmzYle3t7sR7KXfcGBweTjo4O7du3TyGtqvVKSeG6oXB1A1Hx/Rbn950pTgXlGxYWRmvXrqXFixfT2rVrKSwsLN/9qXJtJVVx1mV5qVSpEg0aNEh8r+w86t69O9nb2yukVeUaqTTjJ4efmPwmgf/w4QPWr1+PM2fOAABcXV0xbty4EguwoaOjgw4dOmDv3r348ccf5datWbMG//vf/9C6dWssW7YMLVq0ENdFRUVh5syZOHjwIH7++WeF4CiLFy+Gnp4ewsPDFcbGDRgwAGPHjkXLli2xePFihekWypcvDysrK0mf599//0X//v1Rrlw5cVnOu7GDBg3Ctm3bsGDBArkQw1LTqeL169cFfs73798r3E2OiYlBly5d5PZz7NgxDBgwALt378aHDx/QqFEjbNmyReFphiAIYmQtZZo0aYJXr16pvVtKpUqVFMalFEX37t3V+tS2sDIzM/Hw4UMkJCTgw4cPSrfJ7urq6emJcePGIS4uDra2tpgzZw7279+Pr7/+WtxWW1sbCxcuVLofVc57qezt7TF//vx8u7yvWbMG8+bNkzsP09LSlE5rk1NSUpLSLnhSz3t1KcoxVSVdVlYWGjVqpJ5C55Jfva2O86ion1WVY3rz5k14e3vLnSs56wo3Nzd069YNK1asUIgAqGq9ogquG9RfNwDF91uc33emOOWVb0REBHx9fcVppShHN2oHBwds3bpV7pormyrXVlIVZ12WlzZt2uDQoUN4+PAhqlatqrA+JiYGJ06cEIPp5V4n9RpJiu3bt8PJyUku2n16ejpSU1OVfhdOnjyJkydPynUrLwxuHH5itm/fjm+//RZbt25Fu3btxOVZWVno0qULgoKCxK4BR48exb59+3D27Fno6uqWSPnevXuHxMREheXbtm1DrVq1cPr0aejo6Mita968OQIDA9GwYUNs3bpVoXEYGRmJAQMG5Bk0pWHDhujfvz8OHTqksK5r1644e/aspM/y4cMHWFpaiu8NDAwUogg6Ojrit99+U0s6VVSrVk1urJ8yly5dQo0aNeSWJSYmisFjACAsLAwZGRn48ssvAfzfj1F2ePei0NXVLZbzztPTE3///TfS0tJK1SD7vGRlZWHx4sVYvXq10u9GTtkXLkOHDpWLimhvb48LFy5g1apVuHfvHmxtbTF69Gil81MCqp33UlEhhrcr26Z69eoFdvuMjIxEnTp1FJZLPe9VJeWYqpLOxcVFLrqeuuVVb6tyHkn9rKoe05zBpsqXL6/QFbF27dpKA05pol7huiH/bVSpG4Di/S3O6ztT3HLne/36dXTs2BHv3r1Dhw4dxDGaT548QVBQEE6dOoVOnTohKipKYYiHKtdWUhV3XabM3LlzcejQIbRq1QqLFy8WxxXeuHEDERERmDt3LvT09DB9+nSFtMV5jaSMj48P/Pz85BqHS5YsURgjmS0qKgqrV6/mxuHn7uTJk3jz5o3CXa7ff/8dZ86cQZUqVfD999/D0NAQP/30EyIjI7FhwwZMmDCh2Mt29uxZ7N69G7Vr11ZYd+vWLYwbN06hYZhNR0cHPXr0wNq1axXWvXv3DtbW1vnmbW1trXRuuuXLl6N169aYMGECli5dKncHsSCVK1fG48ePxfe2tra4fPmy3DZxcXEK/eWlplNF9+7dsWbNGgQGBqJ9+/YK6/fu3YuoqCiFCH1GRkZyF09BQUGQyWRwdXUVl+no6Kh10mdVLVq0CJGRkejduzd++OGHIo+xKGmzZ8/G8uXLYWVlBV9fX1SqVEnSsa9RowbWrVtXqG1VOe+L08OHD1GhQgW5ZX369MHChQuxdetWpXduV6xYgWvXrikN5y31vFeV1GMqNd3SpUvRpk0b/Pnnn2qf8yq/eluV80jqZ1XlmFapUgUPHz4U39eoUUNhPNq1a9eU9mbQRL3CdcP/UXfdABTfb3F+35mcHj16pBDkRZV5HpXlu2DBAqSnp+Ovv/5C586d5bafOXMmTpw4AU9PTyxYsAB79uyRW6/KtZVUxVmX5aVBgwb4448/4O3tLUalJSLUr18fRIQKFSpg7969cHBwUEhbFq+RCoMbh5+YS5cuwc3NTWHg644dOyAIAnbu3Cl2P+natStsbW2xf/9+tTQO27Ztq3R5dmjy7MHUyrqO6OrqFvjlefv2rdInTXZ2dvj777+xePHiPNOePn0adnZ2CssHDhwIQ0NDrFu3DgEBAahVq5bSR/OCIOD06dNyy5ydnXHp0iXxfefOnbF69WosWbIEnp6eCAsLw4EDBxQuYKSmU8WcOXPw559/omvXrhgyZAiePHkCAFi/fj0iIyOxe/du2NnZYcqUKXLp6tSpgyNHjmDhwoXQ0tLCrl270KRJE7nogXFxcbC2toZMJoNMJkNMTIwYaa8wEeiUhSdXRaNGjZCWlobo6GicOHEC+vr6sLKyUijL/fv3oaWlpdbyRkZGIjAwMN8Q+bnDb2/btg21a9fGhQsXYGhoWKjPqKWlhYEDB0q+G6nKeV8UuUPxBwcHK90uMzMT8fHx2LNnD5o3by63bvr06di/fz+GDx+OXbt2iX/XGTNmIDIyEhEREXByclI6T5fU8z6nkjqmqqQ7duwYPDw8MGDAALi5uaFx48Z5Hs/cjSZV6m1VziOpn1WVY9qqVSu5p2I9e/bEwoULMWrUKLHuzQ5Cklth65Xnz5+jX79+hf482bhuCFa6XXHVDYD032JVvjMAcOfOHYwZM0Yc4pNTu3btsH79etSsWVNhnZR8g4OD0bdvX4WGYbbOnTujb9++So+lKtdWUhWlLpMaCVbZd83T0xP379/Htm3bcO7cOSQmJsLIyAguLi7w9fWFhYWF0n0V5RqpTNHccEdWHExNTWnmzJlyyzIzM6l8+fJkY2OjsP2QIUPI3NxcLXkLgqD0JZPJyNzcnDp37kynTp1Smtbd3Z0sLCzo0aNHStcnJCSQhYUFeXh4KKybN28eCYJAgwcPVkifkJBAQ4YMIZlMRvPnzy90mZV9htwOHDhAderUofv37xMR0bNnz8jGxoZkMpkYWMfExIT+/fdftaRT1d27d6lly5ZKP1/z5s3F8uT0559/kiAIpK+vTxUqVCCZTEZbt26V28bGxoa8vLzIzc2N3N3dKT4+nohIfF+YlzrZ2tqSnZ1dgS89PT21lffDhw/Ut29f8fhl/5vz/MnrPCpfvjxNnTq1SJ/RxMRE4XteFKqc91Lzyf03UfaqUqUKnT9/XmE/iYmJNGjQINLW1lbY58CBAykxMTHPMkg574lK/piqkk6V46lKva1KvlI/K5H0YxoUFESdO3em2NhYIiJ6/fo1NW7cWO542tvbi+tzKmy9AqDQf5fi+Btx3VD4ukHqb7Eq35kHDx6QtbU1CYJAdevWpVGjRpGfnx+NHj2avvjiCxIEgSpWrEgPHjzI929W2Hz19PRozpw5+R6LOXPmkJ6ensJyVa6tpCrK+Sfle6aOczenolwjqYMgyAdNIlIeQKcw6/LDTw4/MW/fvlXom3/jxg28e/dO6YTAVapUQWJiItq1a4dt27ahatWqed6dyk3IdecwKytLcrmnTJmCnj17omnTppg6dSrc3NxgbW2Np0+fIjg4GKtWrUJiYqLSu8HZXSN+//13/PHHH6hZs6aY9s6dO0hPT0ezZs0wc+ZMhbSqlLlXr17o1auX+N7S0hLR0dHYtGmTOLbD29tboZuI1HSqql69OsLDwxEdHY2oqCi5O2POzs5K0/Tp0wfr1q3Dpk2bIAgCBg4cCB8fH3F9SEgIUlJS0LlzZ4waNUoubV53gotbfuG+86NKeVeuXIn9+/dj6NChGDt2LJo2bYpJkyZhwIABCA0NxdKlS9G+fXulwXkaNmyIhISEIuXXrFkzXLlyRXJ5VTnviyI7FD8RoW3btvDx8VEailtLSwtmZmaoU6eO0uARpqam2LlzJ9asWYMLFy6I566zs3OBd2SlnPdAyR9TVdJJnToDUO1cUCWt1M8KSD+m7u7uckMuDA0NERUVhUOHDolzFfbo0UNpt9LC1ivqnLOV64birRuk/har8jfy9/fHs2fPsH79eowaNUrhyfOvv/6KMWPGYMGCBdi4caPK+VauXBlRUVH5bnPu3DlUrlxZYbkq11ZSFaUuU+cTS6mKco1UphS93cpKsypVqlCvXr3klm3atIkEQaCVK1cqbD958mQCQDKZjP777z8iKrk7h7mtXLmSdHR0xLt2Oe/e6ejo0KpVq/JMm5qaSv7+/lSjRg25MtaoUYMWLFhAqampai0rKxtevHhBp0+fpgMHDtDp06fpxYsXxZJPgwYN5MJZ5767d+3aNSpXrhxt2LBBIe3Ro0fJwMCALl68WOj8IiIiSFdXl7Zt26ZawYugc+fOdODAAcrIyJCU3s/Pj4KDg4ucrnXr1vTbb7/Rq1evJOUrVUkfU1XSlUWa/qyq1A0lVa+UlbqBSLX6oazVDaqoWrUqeXp65ruNp6cnValSJd9tCnsOTpw4kWQyGX377bf0/v17uXXv37+nefPmkUwmy3P6CynXVqr+VhS3kJAQCgkJEf8e2e8L89K0knpyKBAVIkwUKzP69euHI0eOIDo6GnXq1AERwc3NDeHh4bh8+bJchCMA6NixI168eCHX716qnBHSikLI0f/73r172Llzp9wg7UaNGmHQoEGoXr16ofb3+vVrMW3uAey55R73kF8ZlQU4SE9Px8GDB3HhwgUkJSUpjRaV8/Opmq6w1HEssmVkZODnn3/G7t27cfPmTbx7904ccxcdHY3ffvsNkyZNwtKlS9WWZ06ZmZl48eKF0vFewP9NGJ1TbGwsJk6ciGPHjilM3Nu9e3f89NNPhT72hSlvuXLlMHz4cKxZswbAx7vdc+bMwffffy9uM3DgQNy8eRPR0dEK+9y7dy/Gjx8PT09PODo65hmePXuw/IIFCxAeHo7AwEA0btxYvEue+y50XuetlPM+e0ymlZUVfHx8MGzYMKXjYgoSHh6OgIAAhe/44MGD0bp1a4XtdXR0kJWVBV1dXXTr1g3e3t7o2rWr0uBV6jzvS/qYqpqusN/TWrVqyaVT5W+W39RJubdVdh4W5rNu2rRJ0nmWV71SmLohrycSUtNKPTZA2agbAPXUD8VZN+RFym+xKt+ZHTt2YNq0aVi0aFGe282dOxcrVqxQ+ptX1HPw5cuXcHFxwf3792Fubo5mzZqJT/8uXLiA58+fo3r16jh//jzMzMzyLX9hr61UPRdU+b4UJu2GDRsgCAJu3LhRpHgDAJSeH6qUt6hkMhmGDx+O4cOHi8s2btyILVu2ICoqSqHnYPa6Ik/XVOTmJCvVwsLCSCaTkYmJCfXq1YscHR1JEASlE96+fv2aDAwMaPTo0YXad1ZWFt26dUtpX3giEp8m5tUXPL/lUvPM/syTJ0+mx48fK12fkJBAkydPpsjISKVlLky/dmVljI2NJQcHhwL7vudOKzVdURS1n35en/Pdu3fUunVrEgSBrKysqEqVKnLbJCUlkb6+Ps2dO1dteWb7559/qGPHjqSvr6/wNDn7paWlpZDuzp074piOWrVqka+vL82aNYt8fX2pdu3aJAiCuF5d5TU2NpYb52NkZEQjR46U22bGjBlUrlw5hbSpqan09ddfy+Wr7Ol5znxVfbov5by/e/cuzZo1iypXriyua9u2Le3evZvS0tKU5pPbpEmTFD5nzv9PnjxZIc2zZ8/op59+oiZNmojbmZub09ixYyk8PLxQn6ssHFNV0mV/T2UyWYHf09xUqbdVqT8L+1mRxxg+Kce0sHXD3bt31ZZWlWNTVuoGItXrh+KuG5SR+lusynemSpUqBY4/8/LyUvrkUOo5+Pz5c/L19SUDAwO58hgYGNDQoUPp+fPnSssh9dpKlXNBle9LYdO6urqSn58fvXz5koiI5s+fT35+foV65ZWnIBR8jaQOyuqAnN/JwvxeFAY3Dj9BW7duJSMjI7ECaNq0KcXFxSlst3btWhIEgf73v//JLd+/fz95e3vLDeS+f/8+1a9fXzzhBgwYoNBl4N69e9SjRw+ytLSkhQsXUkhICN28eZNCQkLo+++/J0tLS/L09KS7d+9SbGys3EtZnrGxsQXmSUTUu3dvqlmzZr5/EwcHB+rfv7/C8uDgYKWvQ4cO0fz588nU1JT69euntMtLr169SBA+DtYODg6mO3fuKHyu7Jc60hVF7v0U5djk9O2335IgCLRs2TLKysqi+fPnK1Q0nTp1oqZNm6otTyKiy5cvk4GBAVWoUIE8PT1JEARycnKizp07k5WVFQmCQB4eHuTj46OQtk+fPiSTyejXX3+lrKwsuXVZWVm0YcMGkslk1KVLF7WVt379+jRo0CDxfbNmzahOnTpy+bdp04aqVq2qkHbs2LEkCAI5OjrS999/T5s2baKAgAC518SJE2np0qVimrzOW2UvZVQ57zMyMujgwYPUvXt30tbWFi/IJk+eTNevX1eaHxFRQEAACYJAtWvXpl27dtHjx48pKyuLnjx5Qrt376Y6deqQTCbLtzvcjRs3aM6cOWRnZyf+6NWoUYP8/Pzo9u3baj0Hi/uYZr/Uka4o39PcVKm3VTmPCvtZV6xYobZjWti6oW/fvmpLq8qxKWt1A5G0+qEk6gZlpP4Wq/KdGTp0KMlkMtq0aZPSMm3evJm0tLRo2LBhCuuknINxcXGUnJxMRETp6en077//UlhYGP3777+Unp5OREQpKSlKrxFVubYiknYuqPJ9kZo2598oL8nJyUr/RqqUVwofHx9Jr6LixuEn6t27d3T+/HmlP5DZ7t+/T9HR0Qp3ctq3by831obo450sQRCoXbt25OTkRDKZjH777Te5bZYsWUJWVlaUkJCgNL+HDx+SpaUlLVu2TGGd1DyJiKpVq1bgyT9s2DCl0VoLcuvWLTI0NKQ9e/YorDM2Nqb27dsXeZ9S06lC6rFxcHCgtm3biu+V9V8fM2YMWVlZqS1Poo8/SgYGBhQTE0NE8v3s3717R2PGjCFLS0ulUQlNTEwKvDPr6elJJiYmaivvN998Q1WrVhV/sFeuXEmCIFDXrl1p7dq1NHDgQJLJZDR8+HCFtJaWltS0aVP68OFDnuWVyWRy4wzs7e1p9erV+X5GVeR33ueUkJBACxcupBo1aog3cVq1akUBAQEK41tcXFyoWrVqlJSUpHRfSUlJVK1aNXJxcSlUGYODg2n48OFkYmKS51Pk0nxMlZGaTlPf04Lkdx5J/ayqlFdq3aBKWlWOTVmuG4gKXz9oom4gkv5brMo5GBcXR5aWliSTyah+/fr0zTff0IIFC+ibb76hBg0aiE+9lPWWknIOymQyWrBgQb5pFi5cqPTpkjqvrQp7LqjyfZGatrB/I2XnkSrlLc0UQz+xT4KBgQGcnZ3zHadnZ2cHR0dHhbkDY2Ji0KxZM/H969evcezYMQwYMACBgYE4f/486tatiy1btsil27x5M/r3749KlSopza9KlSro37+/QgQuVfIEgGfPnhUY2bNixYp49uxZvtso4+DggF69eikdT5eVlYVGjRoVeZ9S06lC6rF58OABmjZtmu++K1SogOTkZLXlCQBhYWHw9PRE3bp1xWX0//vSGxgYYO3atahcuTLmzJmjkDYzM7PACarr16+v0AdflfIOHToU3bt3x6NHjwAA48ePR/fu3XH8+HGMHz8ef/zxB5ydnZWeR6mpqfDw8Mh3omUdHR18+PBBfB8bG4ukpKR8P6Mq8jvvc6pUqRJmzpyJJUuWoFKlSiAiREREYOjQoahatSqWL18uRti7fv06+vTpA2NjY6X7MjY2Rp8+fXD9+vVCldHNzQ2zZ8/G6NGjoa2trTDWAijdx1QZqek09T0tSH7nkdTPqkp5pdYNqqRV5diU5boBKHz9oIm6AZD+W6zKOWhjY4Pw8HC4ubnh+vXrWL9+PebPn4/169fj2rVrcHd3x9mzZ1GtWjWFtFLOQfr4ECjfNHmtV+e1VWHPBVW+L1LTFvZvpGwbVcpbmvFUFp+QQYMGoW/fvujdu7dK6RMTE1GxYkVxeVhYGDIyMvDll18C+Phj1KFDB4VJdh8+fAh9ff1889DX18fDhw8VlkvNEwBMTEzw4MGDfPONi4sr0kTLOVlZWeG///5TWO7i4oIbN24UeX9S06lC6rGpUKFCgRX/3bt3YWlpqbY8ASA5OVnuxoaOjg7evHkjvpfJZHB3d8fu3bsV0jZu3LjAi4jr168rVOiqlLdx48b45Zdf5Mp7+PBh/PPPP2KI/GbNmikNxd6kSRPcuXMn33xtbGwQFhaGzMxMaGlpAUChB9BLldd5n+3WrVvYtGkTtm/fjufPn0NPTw/e3t4YMmQILl26hLVr12LWrFl49uwZli9fXqg8C/OZEhMT8ccff2DHjh1iiHYjIyOlE4+X5mOqjNR0mvqeFkZe55HUz6rqMZVSN6iSVpVjU1brBqBo9UNhqLtuAKT/Fqv6nXFwcMCZM2cQHx8vF3zHyclJaaMwmyrnb34ePnyoNMCMuq6tinIuqPJ9USVtQfL6GxVnnrmp6xq/UOk18rySFYucXe9USW9lZSUXpGb69OmkpaUlNx5w+vTpCoEYatasSTVq1FDoTpbt7du3VL16daV92KXmSUTUs2dPMjQ0zDNoTVxcHBkaGhYYPlqZ1NRUqlevHlWqVElh3T///EPlypWjffv2FWmfUtMVRkhICIWHh1NmZqbccqnHpnv37mRubi6GCs/dZeLBgwdkYGAgNzZL1TyJPnZnyXk+2NjYUM+ePeW2GTVqFBkaGiqkDQsLIz09Pdq4caPSfH/99VfS19dXCFigSnlVERERQQYGBnTkyJE8t5k/fz4JgkBGRkZkb29PgiCQqakp2dvb5/uqXr26pDLldd6/f/+etm/fTm3atBEHu9etW5d+/PFHhcmmU1NTydXVVexOk9117PXr10rzTElJIRsbG6Vdx9LS0mjfvn3Us2dP0tPTI0EQSFdXlzw9PWnfvn15TlVTmo+pOtNp6ntakPzqT6mfVZXySq0bVEmryrEpS3UDkfT6QRN1A5H032JN1SuFPQeHDRtG/v7+5O/vL47Pz36f8zVv3jwxUI2Hh4fC/lS5tpJ6LqjyfSlK2px/B1X+RqqUt6jUdY1fGPzk8BMTHR2N7du3q7SPOnXq4MiRI1i4cCG0tLSwa9cuNGnSBKampuI2cXFxChPMDh8+HLNnz0arVq0wb948tG7dGubm5nj58iXOnj2LBQsWIDY2FkuWLFFbngAwZcoUHDlyBK1atcLChQvRoUMHVKpUCY8fP8apU6fw7bff4v3795g6dapC2rz+VhkZGXj06BH27NmDmzdvYsKECUpDfHt4eGDAgAFwc3ND48aNlYYZF3KF/D527JikdIXh7u4OQRBgZ2eHGTNmwNfXF7q6upKPzfTp0+Hh4YF27dphzZo1Ynjmd+/eITIyEuPHj0dGRgamTJmiUBZVzocvvvhC7s50q1atcPDgQURGRqJFixa4ceMG9u7dizp16iikPX36NDw8PDBq1CisXLkSrVq1EkN3h4eH49atW+jUqRMCAwMRGBgoprO3t0dgYGChyqvOKRP+/vtvuLu7o2fPnmjbtq3ScPVEhHbt2iE1NRUJCQkQBEGl7kJFOe+zjRs3Drt27UJycjJ0dHQwYMAAjBo1Cm5ubkr3paenh06dOiE8PBwAMGrUKAwbNgwtWrSAn58f3NzcYGFhgRcvXiA4OBj+/v54+PChwvds+PDh2L9/P1JSUkBEaNasGby9vTFw4ECYm5vn+/mLcg6W9DHNTpvzOy41naa+p1LOI1U/a1HKm1edXVDdcPr0abRs2VIundR6xcrKCq9evZJ0bMpK3QCoVj8sXbq0ROoGdf2Ga6peKew5mHv4TXBwMIKDg/PMq3Llyli2bJnCcqnXVqqcC6rUZUVJ6+zsLPe3lvo3UqW8hZGZmSkObQDUc41fKJKboKzUySvUdWFe2Wn9/f3pzz//JEEQSF9fnypUqEAymYy2bt0ql5eNjY3CwOjMzEwaOnSoXBmyI1Rl5zF06FCFKFtEJDnPbD/99BNpaWmJeeX8v7a2Nq1duzbPv1l+fw+ZTEZfffUVpaamFjpEeGHCYEtJVxhubm7Upk0bMjc3J0EQxLu7qhyb9evXk46OjtK/k46OTp53MVXJc82aNaSlpSUO+I+OjhantLCwsCAtLS0SBIEOHDig9JhKfRW2vOo6F4pS3pxp1XEHsbDnfc40NWvWpB9++CHP0Oe5hYWFyYX/Hj9+fL5/3wkTJigtq729PX333Xd069atIn3OopyDmj6mqqQj0sz3VMp5pOpnLe3HVNm+pB6bslI3ZKdTpX4oibpBXedDaT8HASi8lEWgPXv2LF2/fl2hp1FOUq6tVD0XpH5fipI2+28QFBREgiCQr6+v5L+RKuUtyM2bN0kQBPGaJ6/vZ0Gv7LSFrRsEogJuL7EyY9u2bSrvw8nJCY6Ojvjll1/Eu1YDBw7EtGnTxG1CQkLg5eWFpUuXYtSoUQr7CAkJwbZt2/Dvv/8iOTkZxsbGcHR0hLe3N9zd3fPMW5U8AeDff//Fhg0bcOHCBSQnJ8PExATNmjXD6NGjUb9+faVp8vqbyWQymJqaokmTJuKg85CQkDzLXpCcd8yKsp+87rQVxrVr1xAWFobRo0fL5S3l2Ny4cQMbNmzAuXPnkJiYCCMjI7i4uGDs2LEFDpCXkueHDx+QmJgIU1NTMWBSREQEFi1ahHv37sHW1hbjx49Ht27dlOYnlZubW6HKGxcXJzmP3BOHSzkf/P394eHhgTZt2kgqQ1HO+2ynT59Gu3btJOWX09mzZ5VOdD1kyBC4uroqbB8WFqZ0AuyiKAvHVJV02Ur6eyrlPMqZX2Ep+6yFKa+66mx17EvKsSkrdQOgnvqhuOsGdZ4P2fsrrfVKtoCAADRq1Ejp0/vCKuq1lTrOBVXqsqKmVfU7o2p583Pv3j20bdsWgiDAz89P8n6yZV/jF4Qbh4wxxhhjjDHGwFNZMMYYY4wxxhjjxiFjjDHGGGOMMW4cMsYYY4wxxhgDNw4/K2lpafDz80NaWlqJpCtreaqSlstbOvNUJS2Xt3TmqUpaLm/pzFOVtFze0pmnKmk/lzxVScvlLZ15qpJWU+VVSnJ8VVbmJCcnEwBKTk4ukXRlLU9V0nJ5S2eeqqTl8pbOPFVJy+UtnXmqkpbLWzrzVCXt55KnKmm5vKUzT1XSaqq8yvCTQ8YYY4wxxhhj3K2UMcYYY4wxxhigrekCMPXJyspCQkICKlSoAEEQFNanpKTI/VtYUtOVtTxVScvlLZ15qpKWy1s681QlLZe3dOapSloub+nMU5W0n0ueqqTl8pbOPFVJW1x5EhFev36NypUrQyYr3DNBgYioyKVgahMbGwt7e3sMGTIEAQEB4vJ//vkHs2fPRnR0NF68eAFHR0dER0fnu6+HDx+iWrVqxVtgxhhjjDHGWJkRHx+PqlWrFmpbfnJYCqWkpKBbt25ITU2Ft7c3LCwsULFixQLTVahQAQDgZjEY2jLdIud7d7x9kdMAQI0fb0tKBwCZr5KkJeR7GowViaAtvbqnjAyJmSr2YCh8WumjHgQdaZ9VleJmpUv8G6lAy8y0xPPMSkqSnFbyeVQWST2ZVPltk5inoKUlOUuVjqnUv5EKdYNMv+jXRqrKSk2XnpiyJKYrY9dIqlS+ZYzU75ugqyMpXQZ9QOj7/WIboTC4cahhVapUwY0bN2BsbCwuO3/+PJ49e4ZFixZhzpw5hd5XdldSbZmupMahTF+/yGmy85NKEKSd7EAZq/gY0zBBUKFxKPkiTkONQ4mfVVl3/MLK0sDFjZYKda9UWZLrbBXOo7JI8mfVQONQUKFxqNJ3XAONQ0ET3xlVrlckNg7L2jXSZ1Q3SP2+CSqeu0X5fePGoYbp6OigTp06cssSEhIAAJUrV9ZEkRhjjDHGGGOfIY5WqmGxsbEQBAE+Pj4APrbshwwZAgDw9fWFIAgQBEFuPCJjjDHGGGOMqRs/OSxl5s+fj+joaBw6dAg9e/aEk5MTAIj/MsYYY4wxxlhx4MZhKePn54eAgAAcOnQIXl5e4hNFZdLS0pCWlia+lxL+ljHGGGOMMcYA7lZapi1ZsgTGxsbii6exYIwxxhhjjEnFjcMybPbs2UhOThZf8fHxmi4SY4wxxhhjrIzibqVlmJ6eHvT09DRdDMYYY4wxxtgngJ8cMsYYY4wxxhjjxiFjjDHGGGOMMW4cMsYYY4wxxhgDNw4ZY4wxxhhjjIED0rAcDB8IktI9/rKO5Dwr/3lXUrqMZy8k5ynTlx7EJ+vdO8lpNUKQdkxBpN5yMI2jjAwNZKrCeUSZ0pOmSUtLUr8vACCocK81S1p5M19IrwelllemqyM5S42cg5oi9dxX5RyUmCdlSv+uqULQ0pKUTpXySv4NV+G4CNoqfGekflQV6k+N+IyuOaTWg1LTZdGHIqfhxqGG2dnZgXJ9KXx8fODj46OZAjHGGGOMMcY+S9ytlDHGGGOMMcYYNw4ZY4wxxhhjjHHjMF+BgYEQBAFjx45Vuv7u3buQyWTo1KmTuCwuLg7Dhg1DlSpVoKuri6pVq2LYsGF48OCBQno7OzvY2dkp3be7uzsEVcYeMMYYY4wxxlgRcOMwH+3atUONGjWwa9cuvFMyiHnTpk0gIowYMQIAcOvWLTg7O2PLli1o0qQJpk6dikaNGmHLli1o2rQpbt26VdIfgTHGGGOMMcYKhRuH+RAEASNHjkRycjL27dsnty4jIwPbtm2DlZUVevbsCQAYPXo0nj9/jl9//RWHDx/GkiVLcOTIEaxbtw7Pnz/HmDFj1Fq+tLQ0pKSkyL0YY4wxxhhjTApuHBbA19cXurq62LRpk9zyY8eO4fHjxxgyZAh0dHTw4MEDBAUF4YsvvhCfJGYbPXo06tSpgzNnziA+Pl5tZVuyZAmMjY3FV7Vq1dS2b8YYY4wxxtjnhRuHBbC0tETv3r0RFhaGmzdvisuzG4vDhw8HAERHRwMA3NzcFMYKymQytGnTRm47dZg9ezaSk5PFlzobnowxxhhjjLHPCzcOC2HUqFEA/q9BmJCQgOPHj8PNzQ21atUCALFLp7W1tdJ9VKpUSW47ddDT04ORkZHcizHGGGOMMcak4MZhIbi7u6NOnTrYvn070tPTsXXrVmRmZsp1H81umD19+lTpPp48eSK3HfDxiWJGRobS7ZOTk9VVfMYYY4wxxhgrEDcOC2nkyJF4/vw5Dh48iC1btsDU1BR9+vQR1zs5OQEAQkNDQURyaYkIoaGhctsBgKmpKZ49e6bQQHz79i1u375dPB+EMcYYY4wxxpTgxmEhDRkyBPr6+pg8eTLu3bsHb29v6Ovri+ttbGzg4eGB69evY8uWLXJpf/vtN9y4cQNt27aVCxrj7OyMDx8+YOfOneIyIsLs2bPx9u3b4v9QjDHGGGOMMfb/CZT7MRfL0+DBg/H7778DAK5evYr69evLrf/vv//QunVrvHz5Ep6envjiiy9w/fp1HD58GJaWlggLCxPHKALAtWvX0KRJE2RlZWHAgAGwtLTE2bNnkZSUBENDQ1y5ckXhKWR+UlJSYGxsjHZWw6Et0y3y53vcr2aR06iq8p93JaXLePZCcp4yfT3JabOUzHdZquUKjlRoXC2wz5HU7wsACCrca83KlJhnyZdXpqsjOcus1FTJaT8bqhxTqfW2JvIEIGhrS8syU+L3BdDI30jQlv6dkfxZpdYp7JOTQR8QjENITk4udGwSbhwWwenTp9G+fXs0b94ckZGRSreJi4uDv78/Tpw4gefPn8PS0hKdO3fG/PnzYWtrq7B9UFAQZs+ejejoaBgaGqJr165YsWIF+vfvj5CQEEmNQ3dZb2gLRa+MBC2tIqcBgGph0iu+2wu+kJRO79gFyXlq6oeQMcYYY4yxkiKlcSjtts1n6vLlywCgMI9hTra2tgrdSvPj4eGBqKgoheXBwcFFLh9jjDHGGGOMScVjDgspNTUVa9euhampKQYOHFhs+cTGxkIQBPj4+BRbHowxxhhjjDGWGz85LEBYWBhCQkJw8uRJxMXFYcmSJShXrpymi8UYY4wxxhhjasWNwwIEBgbC398fFhYWmDx5MqZNm6bpIjHGGGOMMcaY2n123UpfvXoFLS0tdO/eXW55dHQ0BEGAIAi4c+eOuNzPzw9ubm548+YNlixZgqysLPz888/o1KkTqlWrBj09PVhZWaF3797imMScAgICIAgCAgICcOrUKbRs2RLlypWDubk5hgwZgpcvX8pta29vDwDYtm2bWB5BEHgMImOMMcYYY6xYfXZPDk1NTeHo6IizZ88iMzMTWv8/QmdQUJC4TVBQEGrW/DitQ2pqKqKiotCyZUvo6enhyZMnmDRpElxdXdG1a1eYmpri3r17OHz4MI4fP47Q0FA4Ozsr5Hv48GEcO3YMPXr0QMuWLREaGort27fj7t27CAsLAwA4OTlh4sSJWL16NRwdHeHl5SWmt7OzK74/CmOMMcYYY+yz99k1DoGPEUIvX76MixcvolmzZgA+Nghr1aqF9+/fIygoSIxIGhERgbS0NHh4eAD42Lh88OABqlSpIrfP69evo3nz5pgzZw7+/vtvhTyPHDmC4OBgtGrVCgCQmZmJ9u3bIzg4GFFRUWjevDmcnJwwadIkrF69Gk5OTvDz88v3c6SlpSEtLU18n5KSIvlvwhhjjDHGGPu8fXbdSgGIDb0zZ84A+NhQCw0NhYeHBzw8PBSeIgKAu7s7AEBPT0+hYQgA9erVg4eHB0JDQ/HhwweF9YMGDRIbhgCgpaWFIUOGAAAuXJA2Z9+SJUtgbGwsvqpVqyZpP4wxxhhjjDH2WTYO27RpAy0tLbHhd/nyZSQnJ6Nt27bw8PDAkydPcOPGDQAfG4cGBgZwcXER00dHR2PQoEGwsbGBrq6uOC7wyJEjSE9Px4sXLxTybNKkicKyqlWrAgCSkpIkfY7Zs2cjOTlZfMXHx0vaD2OMMcYYY4x9lt1KjYyM0LhxY4SHh+PDhw8ICgqCIAjw8PDAu3fvAHxsFNra2uL8+fNwc3ODrq4ugI/dTNu2bQsA6NixIxwcHGBoaAhBEHDw4EFcuXJFrqtnzjxz09b++OfPzMyU9Dn09PSgp6cnKS1jjDHGGGOM5fRZNg6Bj11LL1y4gPPnzyM4OBj16tWDpaUlAMDe3h5BQUFwcHDAhw8fxG6oALBo0SKkpaXh7NmzaN26tdw+o6KicOXKlRL9HIwxxhhjjDGmDp9lt1Lg/8Ydnjp1CmfPnhWfBgJA27ZtERwcLI5JzB5vCAB3796FmZmZQsPw3bt3uHTpksrlyo6eKvVpImOMMcYYY4xJ8dk2Dlu3bg1tbW388ssveP36tVzj0MPDAy9evMDmzZtRvnx5uakpbG1t8erVK1y/fl1clpmZiWnTpuH58+cql8vU1BSCIPD4QcYYY4wxxliJ+my7lRoaGsLZ2RmRkZGQyWRwc3MT12U/VXz+/Dk6deoEHR0dcd348eNx6tQptG7dGv3794e+vj6Cg4Px6NEjuLu7qzxZfXa5QkND4e3tDQcHB8hkMnh7e8PW1lalfTPGGGOMMcZYXj7bxiHwsREYGRmJRo0awcTERFxeuXJl1KpVC7du3ZLrUgoA3bt3x59//onFixdjx44dKFeuHNq2bYv//e9/WLBggVrK9fvvv2Py5Mk4evQokpOTQURo3bp14RuHlAUgq8j50gdpXVkftpceFOfhhgxJ6Wpft5Gc5/taVpLTGvwr7YluxjPFCLaFlsVdjBljjDHGWPETiIg0XQimHikpKTA2Noa74AVtQafgBLlJPBVkFSpISof/196dx0VZ7v8ff92ADLgwuICKEmBaVh7T1NI0Bdw4ZQdSj5rlyfXosWzR0qg8QmZoqS1aZotpdkJzzzSs1DH3pdTK1NQj5XHfYFyHbX5/+HV+TmDCDTgC7+fjMY+a+74+9/WZkWU+XNd9XcCe9+qairv1xZOm+1RxKCIiIiKlXZYzExuLSE9Pz3PnhLyU2XsOrxebzYZhGCQkJHg6FRERERERkatScSgiIiIiIiIqDkVERERERETFYbFKSEhwrXyamJiIYRiuR2pqKgAnTpzg6aefJiIiAovFQnBwMN26dePnn3/2YOYiIiIiIlLWlOnVSotbZGQkqampzJgxgzZt2ritfBoYGMjx48dp0aIF+/btIzIykh49erB//37mzp3LkiVLWLZsGa1atfLcCxARERERkTJDxWExulwMzpgxg8jIyFyL0vTt25d9+/YRHx/Pq6++6jq+dOlSHnjgAfr06cPu3bvx8sp7gNfhcOBwOFzP7XZ7kb8GEREREREpGzSt1EMyMjJITk6matWqvPTSS27n7r//ftq3b8/evXtZu3btVa+RlJSE1Wp1PUJDQ4s7bRERERERKaVUHHrIrl27uHjxInfffTfly5fPdf7yvYrbtm276jXi4+NJT093PQ4cMLcHn4iIiIiIiKaVesjlKaDVq1fP83zNmjXd2uXFYrFgsViKPjkRERERESlzNHLoIQEBAQAcPXo0z/NHjhxxayciIiIiIlKcVBwWM29vbwCys7PdjtevXx8/Pz82b97M+fPnc8XZbDYAGjVqVNwpioiIiIiIqDgsblWqVAHIdT+gr68vDz/8MCdOnCApKcntXEpKCsuWLaNu3bq0bNnyuuUqIiIiIiJll+45LGb169cnJCSEWbNmYbFYqF27NoZhMGTIEMaNG8eqVat45ZVXWLduHffccw+pqanMmTOH8uXL8/HHH191GwsREREREZGipOKwmHl7ezN//nxGjBhBcnIyZ86cAeDRRx8lPDycjRs3Mnr0aBYtWsTq1auxWq3ExcURFBTEfffdx8qVK137Jeab0wk4i/y1XFVmpulQi19GESaSPz7nsgoRbO5bxsu3nOkucy5mX7uRiIiIiEghaVjqOrjnnnuw2WzY7XacTidOp5Pw8HAAqlWrxltvvUVqaioZGRkcP36cOXPmEBwc7NmkRURERESkTFFxKCIiIiIiIioORUREREREpBQVh/PmzaNNmzYEBwfj5+dHSEgI7dq1Y968eW7tFi9eTFRUFFarFX9/f+68804mTpxIVpb7fWg2mw3DMEhISMjVV2pqKoZh0Lt3b7fj4eHhhIeHc/bsWZ566ilCQkKwWCw0bNiQuXPn5pn3gQMHePjhh6lSpQoVK1akTZs2fPfdd4V6L0RERERERAqqVCxIM2XKFAYPHkzNmjV56KGHqFq1KkeOHGHTpk0sWLCALl26ADBx4kSGDRtGlSpV6NmzJxUqVOCLL75g2LBhrF69mvnz52MYRqFyyczMpEOHDpw+fZouXbpw/vx5Zs2aRbdu3UhJSaFDhw6utocPH6ZFixYcPHiQjh07ctddd7Fz507at29PVFRUofIQEREREREpiFJRHH744Yf4+vqybdu2XAu5nDx5EoB9+/YxYsQIgoOD2bJlC6GhoQCMGTOGdu3asXDhQj799FN69epVqFwOHTpEs2bNsNls+Pr6AtCzZ0/atWvHxIkT3YrD+Ph4Dh48yCuvvMKLL77oOv7+++8zcODAa/blcDhwOByu53a7vVC5i4iIiIhI2VVqppWWK1eOcuVybxdQtWpVAD777DOysrIYNmyYqzAEsFgsjBs3DoDp06cXSS5vvPGGqzAEaNu2LWFhYWzevNl1LCMjg9mzZxMcHMywYcPc4vv370+9evWu2U9SUhJWq9X1uPJ1iYiIiIiIFESpKA579OjBuXPnaNCgAc899xxLly7NNYq2detWgDz3DGzRogV+fn5s27at0LkEBgYSERGR63jt2rVJS0tzPd+9ezcXL16kadOm+Pn5ubX18vKiZcuW1+wrPj6e9PR01+PAgQOFzl9ERERERMqmUlEcPvvss3z00UeEhIQwYcIEHnjgAapWrUpcXBz79+8H/v+Uy+rVq+eKNwyD6tWrF8m0TKvVmudxHx8fcnJyXM/T09MBrrqfYV55/pHFYiEgIMDtISIiIiIiYkapKA4Nw6Bv375s3ryZ48ePs2DBAjp37syiRYvo1KkT2dnZrsLp6NGjueKdTidHjx51K668vC69NX9cxRT+f2FXGJeLyGPHjuV5Pq88RUREREREikupKA6vdHnEcPbs2URHR/PLL7+wd+9eGjduDFzaouKPNm7cyMWLF2nUqJHrWOXKlQE4ePBgrvaXp6gWxi233IKfnx9btmzh4sWLbudycnJYt25dofsQERERERHJr1JRHNpsNpxOp9uxzMxMTp06BYCfnx89e/bEx8eHiRMncujQIVe7jIwMRowYAeC2b+Gtt95KpUqV+OKLL1zXgUsjeq+88kqhc7ZYLHTr1o1jx44xYcIEt3Mffvghv/76a6H7EBERERERya9SsZVFXFwcAQEBNG/enLCwMDIzM/nmm2/45Zdf6Nq1K2FhYQCMGzeOYcOG0bBhQ7p160aFChVYvHgxu3fvJjY2lkcffdR1TV9fX4YMGcKrr77KXXfdRWxsLGfOnGHx4sW0adOGffv2FTrvsWPHsnz5cl566SXWrFlD48aN2blzJ0uXLqVDhw58/fXXhe5DREREREQkP0pFcZiUlERKSgqbNm1i8eLFVKhQgZtvvpkpU6bQr18/V7uhQ4dSt25dJk6cyKeffkpGRga33HILEyZM4Mknn8QwDLfrjh49Gl9fXz766CPee+89wsPDGTlyJA8++CDz5s0rdN41a9Zk3bp1DB8+nGXLlvHdd9/RpEkTvvnmG1asWFFiisOcP0yLLYibHjFXZB/r2dh0n5V3XzAdeyLqJlNxgburmO7T68c9pmML829j2h++j/LtD6P/IiIiInJ9Gc4/zseUq0pNTSUiIoLHHnvMtSdi7969mTFjBvv37yc8PBy4NM01KiqKUaNGkZCQcN3ys9vtWK1WIonFx8i95+ONyOsP23jk1wkPFYfpdfxNxQXuPmu6TxWHIiIiIlJQWc5MbCwiPT0937salIp7DkVERERERKRwVBwWUlJSEjt37qRWrVqeTkVERERERMS0UnHPoSfVrFmTmjVrejoNERERERGRQtHIYSH17t0bwzBITU29Ztv09HTatGmDl5cXkyZNch0/c+YMo0aN4o477sDf35/AwEA6duzImjVrijFzERERERGR/08jh9fJ4cOHiYmJYdeuXSQnJ9O9e3cATp06RevWrdmxYwctW7Zk0KBB2O12Fi1aRFRUFHPmzCEuLs6zyYuIiIiISKmn4vA6+PXXX+nYsSMnT55kyZIltGvXznVuyJAh7Nixgw8++ID+/fu7jiclJdG0aVP++c9/EhMTg18eq3o6HA4cDofrud1uL94XIiIiIiIipZamlRazzZs306pVK86dO8fKlSvdCsMTJ04we/ZsoqOj3QpDgODgYJ577jmOHz/Ot99+m+e1k5KSsFqtrkdoaGixvhYRERERESm9NHJYjFavXs2ECRMICgpi2bJl1KtXz+385s2byc7OxuFw5Lkf4p49l/a327VrF506dcp1Pj4+nqFDh7qe2+12FYgiIiIiImKKisNitHXrVs6ePUuHDh2oU6dOrvOnTp0CYO3ataxdu/aq1zl37lyexy0WCxaLpWiSFRERERGRMk3TSovRE088Qb9+/Zg/fz49e/YkKyvL7XxAQAAAw4YNw+l0XvUxatQoT6QvIiIiIiJliEYOi5GXlxcffPCB678A//nPf/DxufS2N2vWDMMwWL9+vSfTFBERERER0chhcTMMg6lTpzJw4EA+//xzHn74YdcIYo0aNejWrRvr1q3j9ddfx+l05orfuHEj58+fv95pi4iIiIhIGaORw+vAMAymTJmCl5cXU6ZMwel0MmvWLHx8fHj33XfZvXs3w4cPZ+bMmbRo0YLAwEAOHDjAli1b2LNnD4cPH6Z8+fKefhkiIiIiIlKKqTi8TgzD4J133sHLy4t33nmH7t27M2vWLKpUqcK6deuYPHkys2fP5j//+Q85OTnUqFGDO++8k5EjR1KtWjVPp19sci5eNBVXdeb3pvu8ENPIdOypBubi/E/l3qcy37GmIwHDMBfm7W26S+cf7q0VERERkZLBcOY1l1FKJLvdjtVqJZJYfIxynk6nWBnlfE3HFqY4PNTKXNFU6zvzBZP/8p9Mx+Y4HKbiVByKiIiIlGxZzkxsLCI9Pd21EOa16J5DERERERERUXEoIiIiIiIiKg5FREREREQEFYcel5GRwaRJk+jYsSOhoaFYLBaCg4Pp3LkzW7du9XR6IiIiIiJSRqg49LBTp07x9NNP43A4uP/++3nmmWeIjIxk6dKl3HvvvWzevNnTKYqIiIiISBmgrSw8rHLlyvz+++/UqlXL7fiOHTto3rw5L7zwAt98802esQ6HA8cVq1Ha7fZizVVEREREREovjRx6mMViyVUYAtxxxx1ERUXx3XffkZmZmWdsUlISVqvV9QgNDS3udEVEREREpJRScXgD2LZtGz179uSmm27C19cXwzAwDIPFixeTkZHBiRMn8oyLj48nPT3d9Thw4MB1zlxEREREREoLTSv1sHXr1hEdHQ1Ahw4dqFevHhUrVsQwDBYuXMj27dvdpo5eyWKxYLFYrme6IiIiIiJSSqk49LAxY8bgcDhYvXo1rVq1cju3YcMGtm/f7qHMRERERESkLNG0Ug/bt28fVapUyVUYnj9/nh9++MFDWYmIiIiISFmj4tDDwsLCOH36NDt27HAdy87O5tlnn+X48eMezExERERERMoSTSv1sCFDhvD111/TqlUrunXrhp+fHzabjYMHDxIZGYnNZvN0iiIiIiIiUgaoOCyE6dOn06dPHz7++GN69+5t6hqdOnVi7ty5vPrqq3z66aeUL1+e6OhoFixYwMsvv1y0CZcizswM07H+X5mfrut1TzNTcb938DbdZ72leS9IlC9Op7mw7GzTXRomF0lyXmXhpVLJMMzHmvw3FfE4fd1fm94jEfEwTSv9E6mpqRiGYbrwy68uXbrw/fffc+7cOY4fP87s2bOpU6cO06dPx+l0Eh4eXqz9i4iIiIiIaOSwEB566CGaN29OzZo1PZ2KiIiIiIhIoag4LASr1YrVavV0GiIiIiIiIoVW4qeV2mw2DMMgISGBLVu20L59eypVqoTVauWhhx4iNTXVrf2CBQt4+OGHqVu3LuXLl8dqtXLfffcxb948t3bTp08nIiICgBkzZmAYhutxeZGY6dOnYxgG06dPBy5tP1GpUiVuvvnmq+bbsGFD/P39sdvtrmNOp5Np06bRsmVLAgICKF++PE2bNmXatGmFf4NERERERETyocQXh5dt3ryZ1q1b4+vry8CBA2natCkLFy6kXbt2XLx40dUuPj6eHTt20KpVK5566in+/ve/s3v3brp27cqkSZNc7Ro1asRTTz0FwJ133smoUaNcj6vdA1i+fHm6dOnCf//7X9atW5fr/Pbt2/npp5+IjY0lICAAuFQYPvLII/Tr14/jx4/Ts2dP+vfvz7lz5+jXrx/PPvtsEb5LIiIiIiIieSs100qXLl3KrFmz6N69u+vYP/7xD2bOnMnChQvp0aOHq12dOnXcYs+ePcu9997LyJEj6devH+XLl6dRo0Y8/fTTvPXWWzRq1IiEhIR85dGrVy9mzJjBp59+yr333ut2bubMma42l3344YckJyfTp08fpk6dSrly5QDIyMiga9euTJgwgYcffpgmTZrk6svhcOC4YoXHK0cjRURERERECqLUjBy2bt3arTAE6Nu3L3BpVPGyPxaGABUrVqR3796kp6e7tTUjKiqKWrVq8fnnn5OZmek6npOTw2effUZQUBAdO3Z0HZ88eTIVKlTgnXfecRWGAL6+vowZMwaA5OTkPPtKSkpy3fdotVoJDQ0tVO4iIiIiIlJ2lZqRw7xG1mrXrg1AWlqa69ixY8cYO3YsX331Fb/99hsXLlxwizl06FCh8vDy8uKRRx7htddeY+nSpcTGxgKwfPlyDh8+zJAhQ/DxufS2nz9/np9++omQkBDGjRuX61qXi8tdu3bl2Vd8fDxDhw51Pbfb7SoQRURERETElFJTHF6+h+9Kl4uw7P/b0PvUqVM0a9aM33//nZYtW9KuXTsCAwPx9vZm27ZtLFq0yG2aplm9evXitdde49NPP3UVh3lNKT19+jROp5ODBw+SmJh41eudO3cuz+MWiwWLyQ3HRURERERErlRqisP8+Oijj/j9998ZPXo0L730ktu5sWPHsmjRoiLpp0GDBjRq1Igvv/yS9PR0ypUrx4IFC7j11ltp1qyZq93lgrZJkyZs2bKlSPoWERERERExo9Tcc5gf+/btA3CN5l1p9erVuY55e3sD/3/ksSB69erFxYsXmTt3LgsWLODs2bM8+uijbm0qVarEbbfdxs6dO92mvoqIiIiIiFxvZao4DAsLA2DNmjVuxz/77DOWLl2aq33lypUxDIMDBw4UuK+ePXvi7e3NzJkzmTlzJoZh5CoOAZ588knOnz/PgAED8pw+un///lx7NYqIiIiIiBS1Ak8rjY6OZv78+QQGBrodt9vtxMXFsWLFiqLKrcj16tWLcePGMWTIEFauXElYWBjbt29n+fLldO7cmfnz57u1r1ixIs2aNeO7776jV69e1KtXDy8vL3r16uUqNK+mRo0atGvXjq+//hovLy9atWqV5/6IAwcOZMOGDcyYMYO1a9fSrl07QkJCOHr0KLt27WLjxo189tlnV91bUUREREREpCgUuDi02WxkZGTkOn7x4sU8p2beSGrXrs2qVasYPnw43377LVlZWdx11118/fXXHDhwIFdxCJcWknnmmWdc9w86nU5atWp1zeIQLhWjy5YtIzs7O89RQwDDMJg+fTr3338/H3zwAV9++SVnz54lODiYevXqMX78eNq1a1fo117aGOV8zQc7c0yHWveaizveJvPaja7CJ6Sm6disgyZX33U6TffpLIJFnUq9Qry/eHmbi8sp+PR4ERERKVsMpzN/n1J+/PFHABo1asSKFSuoUqWK61x2djYpKSlMnTq1TEyBvDyKd6O9VrvdjtVqJZJYfIxy1w4owTxVHJ56tNm1G+XheJvcf1DJr9tfOmw61nRxKDcuFYdSUhmG+djC/EGlJNF7JCJFKMuZiY1FpKen57mzQ17yPXLYqFEjDMPAMAyio6Nznff392fSpEn5z1ZERERERERuGPkuDvfv34/T6aROnTps2rSJoKAg1zlfX1+Cg4Ndq3uKiIiIiIhIyZLv4vDyPXY5OVefkud0OjEKMyVCREREREREPKLAW1n07t07zy0XUlNTad26dYETmDdvHm3atCE4OBg/Pz9CQkJo164d8+bNc13XMAx69+7Njh07eOCBBwgMDKRixYp06NCB77//Ps/rnjlzhlGjRnHHHXfg7+9PYGAgHTt2zLWNBcD333/PE088QYMGDbBarfj7+/OXv/yFsWPHkpmZ/4VEJk6ciJeXF23btuXMmTOu44sWLaJt27ZUrlwZPz8/GjRowPjx43Ptnzh9+nTXAjWLFy+mZcuWVKpUSSuVioiIiIhIsStwcbh9+3YaNmzI+vXrXcdmzJjBnXfeSbVq1Qp0rSlTptC1a1f27NnDQw89xNChQ4mJieHIkSMsWLDAre1///tfWrZsyYULF/jXv/7F3/72N1auXEnr1q3ZuHGjW9tTp07RokULXn75ZSpXrsygQYPo0qUL33//PVFRUSxcuNCt/QcffMCCBQv4y1/+wsCBA+nXrx9Op5P4+Hh69OhxzdfhdDoZPnw4w4YNo2vXrnz11VdUqlQJgPj4eOLi4ti9ezedO3dm8ODB+Pv789xzz1312nPmzKFz584EBwczePBg/vrXvxbgXRURERERESm4Am9lsWnTJl544QUiIyMZNmwYe/fu5auvvmLixIkMGDCgQNf68MMP8fX1Zdu2bQQHB7udO3nypNvz1atX8/zzz5OUlOQ69thjjxETE8OAAQNcq6kCDBkyhB07dvDBBx/Qv39/1/GkpCSaNm3KP//5T2JiYvDz8wPghRde4J133nG7Z9LpdNK/f3+mTZvG2rVradmyZZ6vISsri379+vHJJ5/w+OOP8/bbb+Pldanm/uabbxg7diwdO3Zk3rx5VKhQwXXtwYMH89577zFv3jy6dOnids2UlBSWLVt2zS0sHA4Hjiu2DbDb7X/aXkRERERE5GoKPHJYrlw5Xn/9dZ5//nnGjh3LwoUL+frrrwtcGF55vXLlcm+7ULVqVbfngYGBvPjii27HOnbsSNu2bfnpp59c00tPnDjB7NmziY6OdisMAYKDg3nuuec4fvw43377rev4TTfdlGsxHcMwePzxxwHc2l7p/PnzxMbG8sknn5CYmMjkyZNdhSHA5MmTAXj//fddheHla48dOxbDMEhOTs513djY2HztbZiUlITVanU9QkNDrxkjIiIiIiKSlwKPHGZmZvL888/zzjvvEB8fz5o1a+jcuTMfffQR999/f4Gu1aNHD4YPH06DBg3o2bMnUVFRtGrVKs99OBo3bkzFihVzHb/vvvtYvnw5W7dupUmTJmzevJns7GwcDgcJCQm52u/ZsweAXbt20alTJwAyMjKYPHkys2bNYteuXZw9e5Yrt388dCj3XnEXLlygbdu2bNq0iffee4+BAwfmarNhwwYqVKjAtGnT8nz9/v7+7Nq1K9fxu+++O8/2fxQfH8/QoUNdz+12uwpEERERERExpcDFYdOmTTl//jw2m43mzZvjdDp57bXX6Ny5M3379uXdd9/N97WeffZZqlatypQpU5gwYQLjx4/Hx8eHBx54gDfeeIOIiAhX2+rVq+d5jcvH09PTgUv3GwKsXbuWtWvXXrXvKxfV6dq1K4sXL+aWW26he/fuBAcHU65cOdLS0njrrbfcpm5edubMGbZu3UrVqlWJiorKs49Tp06RlZVFYmJivvL442u6FovFgsViyVdbERERERGRP2OqOHz77bdd0yQNw2DEiBF06NCBXr16FehahmHQt29f+vbty8mTJ1m9ejXJycl8/vnn7Nmzx+0+wqNHj+Z5jcvHrVYrgGvUcdiwYYwfP/6aOWzevJnFixfTsWNHlixZ4ja9dMOGDbz11lt5xgUHBzN16lTi4uKIjIxk5cqV3HrrrW5tAgICMAyDEydOXDOPK2k7EBERERERud4KfM/hRx995Hb/3GWNGze+6rYS+VG1alXi4uJc9wv+8ssv7N2713V+69atnD17Nlfc6tWrXf0DNGvWDMMw3FZT/TP79u0D4IEHHsh13+Hla19Nx44d+eKLL0hLSyMqKordu3e7nb/nnns4efKkayqriIiIiIjIjarAxSHAzJkzadmyJSEhIfz2228AvPnmm6SkpBToOjabze3ePrh0T+PlqaGXVxMFSEtLY8yYMW5tly1bxvLly2nQoAFNmjQBoEaNGnTr1o1169bx+uuv57o+wMaNGzl//jwAYWFhALn2P9yxY4fbyqhX0759exYvXkxaWhqRkZFu9xA++eSTAK6R0T86cuQIO3fuvGYfIiIiIiIixa3A00qnTJnCv//9b55++mnGjBnj2sg9MDCQN998k9jY2HxfKy4ujoCAAJo3b05YWBiZmZl88803/PLLL3Tt2pWwsDBSU1OBSwvPTJkyhY0bN9K8eXNSU1OZM2cO/v7+fPjhh27Xfffdd9m9ezfDhw9n5syZtGjRgsDAQA4cOMCWLVvYs2cPhw8fpnz58tx9993cfffdfP755xw+fJjmzZvz+++/88UXX/DAAw8wd+7ca76Otm3b8uWXX/Lggw8SFRXFihUruO2224iJiWHkyJGMHj2aunXrEhMTQ1hYGCdPnmTv3r2sXr2aV155hdtuuy3//wAiIiIiIiLFoMDF4aRJk/jggw+Ii4tj7NixruNNmzbl2WefLdC1kpKSSElJYdOmTSxevJgKFSpw8803M2XKFPr16+fWtk6dOkyZMoXhw4fzzjvvkJ2dTWRkJGPHjnWNGl5WpUoV1q1bx+TJk5k9ezb/+c9/yMnJoUaNGtx5552MHDmSatWqAeDt7c2XX37J888/T0pKCps3b6ZevXqMHz+ev/71r/kqDgGio6NZsmQJnTp1chWIt99+Oy+//DKtW7fm7bffZvny5aSlpVG1alUiIiJISEjgkUceKdB7Jv/HmVOI0NyjyflVJfkHU3FBKYGm+zzRvo7p2LRbw0zF1Rn3s+k+Dd/cW9PkR44997Tx/HJmZZqONd+p+a8jkTJJ3zPXpvdISjKza2bo6/6GYjjzmnf5Jy5vvxAWFkalSpXYvn07derUYc+ePTRs2JALFy4UaYKpqalERETw2GOPMX369CK9dmljt9uxWq1EEouPYe4Deklh+BT47xouhSkOjXLm+vWuHGi6z8IVh+biVBzmp1MP/TLz8r52m7zkZBdtHiIiIldScXjDyXJmYmMR6enpeW4VmJcC33MYERHBtm3bch1PSUnR9EgREREREZESqsDDIEOHDuXxxx/n4sWLOJ1ONm3aRHJyMklJSbnu/RMREREREZGSocDFYf/+/fH39+ell17i/Pnz9OzZk5CQEN566y169OhRHDmKiIiIiIhIMTN1A9UjjzzCI488wvnz5zl79izBwcFFnZdLeHh4nttRiIiIiIiISNEp8D2H0dHRpKWlAVC+fHlXYWi324mOji7S5OTPORwO7Ha720NERERERMSMAheHNpuNjIyMXMcvXrzI6tWriyQpyZ+kpCSsVqvrERoa6umURERERESkhMr3tNIff/zR9f+//PILR44ccT3Pzs4mJSWFWrVqFW128qfi4+MZOnSo67ndbleBKCIiIiIipuS7OGzUqBGGYWAYRp7TR/39/Zk0aVKRJid/zmKxYLFYPJ2GiIiIiIiUAvkuDvfv34/T6aROnTps2rSJoKAg1zlfX1+Cg4Px9ja5ObOIiIiIiIh4VL6Lw7CwMABycnKKLRkRERERERHxjAIvSCPXT2RkJIZhYLPZPJ2KiIiIiIiUcioOb2CXR2l9fExtRykiIiIiIpJvKg5vUE6nkx07dhAeHk7z5s09nY6IiIiIiJRyGpK6Qe3YsYNTp04xevRojRzmwZmV5Zl+Hdmm4rKOHDXdZ+Xkk6ZjN/6+xVRcTEJT033mnDljOlbyIcfc16CIiEixcjo9nYEUgQKPHNapU4eTJ3N/WE1LS6NOnTpFkpTA6tWrqV69On379vV0KiIiIiIiUgYUuDhMTU0lOzv3X64dDgcHDx4skqQE/vWvf3HkyBH8/Pw8nYqIiIiIiJQB+Z6v+MUXX7j+f9myZVitVtfz7Oxsli9fTnh4eJEmJyIiIiIiItdHvkcO4+LiiIuLwzAMHnvsMdfzuLg4evTowTfffMOECROKM9fr7ttvv8UwDAYPHpzn+X379uHl5UXHjh1dx3777Tf69etHrVq18PX1pXbt2vTr14/ff/89V3x4ePhVC+rL21iIiIiIiIhcD/kuDnNycsjJyeGmm27i2LFjruc5OTk4HA52795Np06dijPX665t27bcfPPNfPbZZ5w/fz7X+Q8//BCn08mAAQMA+PXXX2nWrBnTpk2jSZMmDBs2jMaNGzNt2jSaNm3Kr7/+er1fgoiIiIiISL4U+J7D/fv3U61ateLI5YZjGAb//Oc/SU9PZ86cOW7nsrKymDFjBsHBwcTGxgIwaNAgjh8/ztSpU/niiy9ISkpi8eLFvPPOOxw/fpx//etfRZqfw+HAbre7PURERERERMwo8B4JL7/88p+e//e//206mRtRnz59GDlyJB9++CGPPfaY6/iSJUs4fPgwzz33HOXKleP3339n5cqV3H777a6RxMsGDRrEpEmTWLFiBQcOHCA0NLRIcktKSiIxMbFIriUiIiIiImVbgYvDBQsWuD3PzMxk//79+Pj4cPPNN5e64jAoKIjOnTsza9Ysdu3aRf369YFLU0oB+vfvD8C2bdsAaNOmTa57Bb28vGjdujW7du1i27ZtRVYcxsfHM3ToUNdzu91eZNcWEREREZGypcDF4datW3Mds9vt9O7dm4ceeqhIkrrRDBw4kFmzZvHhhx8yfvx4Dh06xFdffUWbNm245ZZbAFxTOqtXr57nNWrWrOnWrihYLBYsFkuRXU9ERERERMquAt9zmJeAgAASExMZOXJkUVzuhhMZGUn9+vX55JNPyMjI4OOPPyY7O9tt+mhAQAAAR48ezfMaR44ccWsHl0YUs7Ky8myfnp5eVOmLiIiIiIhcU5EUh3CpmCnNBc0///lPjh8/zsKFC5k2bRqVK1emS5curvONGjUC4LvvvsPpdLrFOp1OvvvuO7d2AJUrV+bYsWO5CsRz586xZ8+e4nkhIiIiIiIieSjwtNK3337b7bnT6eTw4cPMnDmTv/71r0WW2I3mscce44UXXuCZZ57h0KFDPPnkk/j5+bnO33TTTURFRbFy5UqmTZtGv379XOfef/99du7cSXR0tNs9gc2aNeOHH37gP//5j2uxG6fTSXx8POfOnbt+L05ERERERMo8w/nHYa5riIiIcHvu5eVFUFAQ0dHRxMfHU6lSpSJN8Ebyj3/8g5kzZwLw008/0aBBA7fzu3fvplWrVpw8eZK//e1v3H777ezYsYMvvviCoKAg1qxZ47pHEeDnn3+mSZMm5OTk0L17d4KCgli9ejVpaWlUrFiR7du35xqF/DN2ux2r1UoksfgY5YrmRYvHGT4F/huOS8rvW0zFxdzU1HSfzqtMlRYRERGR6yfLmYmNRaSnp7vd2vZnCvypc//+/QVOrLR47LHHmDlzJs2bN89VGALceuutbNmyhcTERFJSUliyZAlBQUH06dOHUaNGERYW5ta+QYMGpKSkEB8fz9y5c6lYsSL3338/48ePp1u3btfrZckNzpmdbTp2Z8Z5U3HetWqa7jPr9/+ZCyzY36lEREREpIiZH5IADhw4AFBmtk+4vFLrH/cxvFJYWBjTpk3L9zWjoqLYsGFDruM2m63A+YmIiIiIiJhV4AVpsrKyGDlyJFarlfDwcMLDw7Farbz00ktkZmYWR443hIsXLzJ58mQqV65Mjx49iq0fwzCIjIwstuuLiIiIiIjkpcAjh0OGDGH+/Pm89tprtGjRAoD169eTkJDAyZMnmTJlSpEn6Ulr1qxh1apVLFu2jN9++42kpCTKly/v6bRERERERESKVIGLw88++4xZs2a5rUzasGFDQkNDefjhh0tdcfjtt9+SmJhItWrVeOaZZ3j22Wc9nZKIiIiIiEiRK/C0UovFQnh4eK7jERER+Pr6FkVOxcZms2EYBgkJCaxbt46oqCgqVapEUFAQgwcP5sKFCwAsWbKEFi1aUKFCBaZMmcJzzz3H4cOHmThxIj4+PiQkJGAYRp73BU6fPh3DMJg+fbrb8ZUrV/LXv/6VkJAQLBYL1atX57777uP99993yw1g1apVGIbhevzxWiIiIiIiIkWtwMXhE088wejRo3E4HK5jDoeDMWPG8MQTTxRpcsVl48aNtG3bFqvVysCBA7npppuYMmUKAwYMYPbs2XTt2pWwsDAGDhxIYGAgr7/+Oq+++qrp/pYsWULbtm3ZuHEjHTt2ZNiwYfztb3/D4XC4tsYIDw9n1KhRwKVFbUaNGuV6NGrUqChetoiIiIiIyFUVeFrp1q1bWb58ObVr1+bOO+8EYPv27WRkZNC2bVs6d+7sajt//vyiy7QIpaSksHDhQmJjYwHIzMykadOmfPbZZyxbtozvvvuOZs2aAZCYmEjdunV56623iI+Pp1y5gu8fOG3aNJxOJytXrnS9Z5edPHkSuFQcJiQkkJiY6Pr/a3E4HG5Fut1uL3BuIiIiIiIiYKI4DAwMpEuXLm7HStpWFlFRUa7CEKBcuXJ07dqVH3/8kQcffNBVGAJUqlSJTp06MW3aNP73v/8RERFhul9/f/9cx6pWrWr6eklJSSQmJpqOFxERERERuazAxeHHH39cHHlcV3lN06xZs+Y1zx06dMhUcdijRw/mz59P8+bN6dmzJ23btuW+++6jWrVqBb7WleLj4xk6dKjrud1uL3GFuoiIiIiI3BgKfM9hdHQ0aWlpuY7b7Xaio6OLIqdiFxAQkOuYj4/PNc+Z3cfx73//OwsXLuQvf/kL7733Hp07dyY4OJi2bduybds2U9eES4sDBQQEuD1ERERERETMKHBxaLPZyMjIyHX84sWLrF69ukiSutF5eV1627KysnKdS09PzzMmNjaWVatWcfr0ab766iv69++PzWYjJiYmz2JbRERERETkesr3tNIff/zR9f+//PILR44ccT3Pzs4mJSWFWrVqFW12N6jKlSsDcPDgwVzntm7d+qexlSpVIiYmhpiYGLKzs5k2bZprFVO4VHhmZ2cXfdIiIiIiIiJ/It/FYaNGjVz77uU1fdTf359JkyYVaXI3qssL1nzyySf06tXLNZK4fv16/vOf/+Rq/91339GyZUu8vb3djh87dgwAPz8/17EqVarwv//9r7hSFxERERERyVO+i8P9+/fjdDqpU6cOmzZtIigoyHXO19eX4ODgXMVPadW8eXNatmzJihUraNGiBa1bt+a3335j0aJFPPjggyxYsMCt/ZNPPsmhQ4do1aoV4eHhGIbBmjVr2LRpE82bN6dVq1auttHR0Xz++efExcXRuHFjvL29+dvf/kbDhg2v98sUEREREZEyJN/FYVhYGAA5OTnFlkxJsmjRIoYOHcqXX37JTz/9xJ133snixYtZsWIFAAsXLqR3797ApVVF58+fz/fff8+yZcsoV64c4eHhjBs3jsGDB7sV1W+99RYAK1asYPHixeTk5FC7dm0Vh2WZ02k69NnmD5mK+yWxhuk+a6wyN73c+vkW0316VzO/JUzOmbPm4s6dM92niIiIyI3IcDoL9snzk08++dPz//jHPwqVUElns9mIiopi1KhR+drIvijZ7XasViuRxOJjlLuufcuNyadGdVNxvyTeZLrPGqvMzSBQcSgiIiJSdLKcmdhYRHp6er53NSjwPodPPfWU2/PMzEzOnz+Pr68v5cuXL/PFoYiIiIiISElU4K0sTp8+7fY4e/Ysu3fvplWrViQnJxdHjiIiIiIiIlLMClwc5qVevXqMHTs216hiWbdmzRoiIyOpVKkSgYGBdOnShb179+Zqd+zYMZ555hnq1q2LxWKhWrVqdOnShZ9//tkDWYuIiIiISFlUJMUhgI+PD4cOHSqqy5V4GzZsoG3btlitVoYMGUKbNm1YsGAB9957L//9739d7fbt20eTJk148803ufnmmxkyZAj3338/KSkpNG/enI0bN3rwVYiIiIiISFlR4HsOv/jiC7fnTqeTw4cPM3nyZFq2bFlkiZV0y5Yt47333mPgwIGuY1OnTmXQoEE89dRTLF68GLi0gM/hw4dJSUmhY8eOrrYvvfQSTZs2ZcCAAfz444959uFwOHA4HK7ndru9mF6NiIiIiIiUdgUuDuPi4tyeG4ZBUFAQ0dHRTJgwoajyKvFuueUWBgwY4HZswIABTJgwgSVLlnD8+HH+97//sW7dOvr27etWGF4ZP3HiRH7++WcaNGiQq4+kpCQSExOL9XWIiIiIiEjZUODiUPsc5k/Lli3x8nKftevl5UXLli3Zs2cP27dvZ8+ePQAcPXo0z20vdu3a5fpvXsVhfHw8Q4cOdT232+2EhoYW4asQEREREZGyosDF4WUnTpwAoFq1akWWTGlSvXre+8tdPp6ens6pU6cAWLJkCUuWLLnqtc5dZT81i8WCxWIpZKYiIiIiIiIFXJAmLS2Nxx9/nGrVqlG9enWqV69OtWrVeOKJJ0hLSyumFEumo0eP/ulxq9Xq2oxy0qRJOJ3Oqz4ee+yx65a3iIiIiIiUTfkeOTx16hQtWrTg4MGDPPLII9x2220A/PLLL0yfPp3ly5ezbt06KleuXGzJliRr164lJyfHbWppTk4O69atwzAM7rzzTldxuH79ep544glPpSoiIiIiIpL/kcOXX34ZX19f9u3bx9SpU3n66ad5+umnef/999m7dy/lypXj5ZdfLs5cS5Rff/2VDz74wO3YBx98wK+//soDDzxAUFAQd999N/fccw/JycnMnj071zVycnJYtWrV9UpZRERERETKsHyPHC5cuJCpU6fmeS9djRo1eO211xg0aBBvvPFGkSZYUnXs2JEnn3ySpUuXcscdd7Bjxw4WL15MtWrVeOutt1ztkpOTiYqKokePHrz55pvcdddd+Pv78/vvv7N+/XqOHz/OxYsXPfhKRERERESkLMj3yOHhw4e54447rnq+QYMGHDlypEiSKg2aN2/O8uXLSU9P5+2338ZmsxEXF8f69eupU6eOq11ERARbt27lpZde4uzZs3z88cdMnTqVbdu20bp1a5KTkz34KkREREREpKwwnE6nMz8Na9WqxezZs2nVqlWe51evXk337t05dOhQkSZYHFJTU4mIiOCxxx5j+vTpruO9e/dmxowZ7N+/n/Dw8AK39TS73Y7VaiWSWHyMcp5OR0owr/LlTcdGbjxuKs7WPNh0n3gVaG0tN0Z1cysuZ+9LNd0n+fuxKyIiImJaljMTG4tIT093rXVyLfn+RNWxY0defPFFMjIycp1zOByMHDmSmJiY/GcrIiIiIiIiN4x833P48ssv07RpU+rVq8fjjz9O/fr1cTqd7Ny5k3fffReHw8HMmTOLM9ciU6tWLXbu3InVajV9jaSkJJ5//nlq1apVhJmJiIiIiIh4Rr6Lw9q1a7N+/XoGDx5MfHw8l2ejGoZB+/btmTx5MqGhocWWaFEqV64c9evXL9Q1atasSc2aNYsoIxEREREREc8q0I06ERERfPXVV5w4cYINGzawYcMGjh8/TkpKCnXr1i2uHHM5ffo03t7edOrUye34tm3bMAwDwzDYu3ev27nIyEj8/f1xOBykpqZiGAa9e/c2nUPv3r0xDIPU1FTXMZvNhmEYJCQksGXLFtq3b0+lSpWwWq089NBDbm2vNH/+fJo2bYq/vz/Vq1dnwIABnD59mvDw8BvmfkYRERERESndTK3iULlyZe6++27uvvtuqlSpUtQ55av/O++8k9WrV5Odne06vnLlyjz//+LFi2zYsIEWLVpgsViKPb/NmzfTunVrfH19GThwIE2bNmXhwoW0a9cu17YU06ZNo0uXLuzZs4d//OMfPPbYY6xfv5727duTmZlZ7LmKiIiIiIiAyeLwRhAVFYXdbuf77793HVu5ciW33HILoaGhbsXhunXrcDgcREVFXZfcli5dyscff8ySJUsYP348y5cvp1evXuzbt4+FCxe62qWlpfHUU09RoUIFtmzZwtSpU3nttdfYtm0bVqv1miu/OhwO7Ha720NERERERMSMEl0cAqxYsQKA7OxsvvvuO6KiooiKispzFDEyMvK65Na6dWu6d+/udqxv377ApVHFyxYtWsTZs2fp168f9erVcx338fHhlVdeuWY/SUlJWK1W16Ok3PMpIiIiIiI3nhJbHLZu3Rpvb29X4bd161bS09OJjo4mKiqKI0eOsHPnTuBScejv788999xzXXJr0qRJrmO1a9cGLo0WXrZ9+3aAPPeOvOeee/Dx+fP1guLj40lPT3c9Dhw4UIisRURERESkLMv3aqU3moCAAO666y7Wrl1LZmYmK1euxDAMoqKiOH/+PHCpKAwLC2PTpk20adMGX1/f65bbH10u9K68R/LyNNDg4Nybf3t5eVGt2p9vzm2xWK7LPZQiIiIiIlL6ldiRQ7g0tfTcuXNs2rQJm83GHXfcQVBQEGFhYURERLBy5UpX8Xi97jcsiMtF5LFjx3Kdy8nJ4cSJE9c7JRERERERKaNKfHEI8PXXX7N69Wqio6Nd56Kjo7HZbK57Eq/X/YYFceeddwKwdu3aXOc2bdpEVlbW9U5JRERERETKqBJdHLZq1QofHx+mTJnCmTNn3IrDqKgoTpw4wUcffUSFChVo1qyZBzPNW2xsLBUrVuSjjz5i3759ruNZWVmMHDnSg5mJiIiIiEhZU6KLw4oVK9KsWTOOHz+Ol5cXbdq0cZ27PKp4/PhxWrVqRbly5TyV5lUFBgYyceJEzp49S5MmTRg0aBAjRoygcePGnD59mpCQELy8SvQ/kYiIiIiIlBAldkGay6Kioli/fj2NGzcmMDDQdTwkJIRbbrmFX3/99YacUnrZgAEDqFy5Mq+++irTp0/HarXyt7/9jXHjxhEWFsbNN9/s6RSlDHJmZJiOXdX5L6biHC2rmO7T+7z5KdjeF03GNmtguk+vn/ddu9FV5Fy4YCrO8CnEH8icOebCrliAq+DBTvOxhmE+1qzC5CsicqMoxM9P079nTP6OKYxC/X4qhOv9HhlOJxTwY47hdOo32o1o79691KtXj27dujF79ux8xdjtdqxWK5HE4mPceCOlUnIY19hG5c94RdxkKu5iRMkqDp1e5n+BqjjMT7CKQxGR607FYbG63u9RljOTlVnzSE9Pz3M3hbxozmIxs9lsGIZBQkJCnudPnz6Nw+FwO3bhwgWeeeYZAOLi4oo5QxERERERkVIwrbSkW7VqFf369aNDhw7cdNNNnDhxghUrVpCamkp0dDTdu3f3dIoiIiIiIlIGqDj0sDvuuIP27duzdu1aFi5cCEDdunUZPXo0zz77rBakERERERGR60KVRzFKSEhwrZqamJiIYRiuR2pqKgCVK1emRo0aeHt7k5OTQ6VKlbjtttuIi4vDz8/Pg9mLiIiIiEhZopHDYhQZGUlqaiozZsygTZs2bqumBgYGcvz4cVq0aMG+ffuIjIykR48e7N+/n7lz57JkyRKWLVtGq1atPPcCRERERESkzFBxWIwuF4MzZswgMjIy16I0ffv2Zd++fcTHx/Pqq6+6ji9dupQHHniAPn36sHv37qtOLXU4HG6L2djt9iJ/DSIiIiIiUjZoWqmHZGRkkJycTNWqVXnppZfczt1///20b9+evXv3snbt2qteIykpCavV6nqEhoYWd9oiIiIiIlJKqTj0kF27dnHx4kXuvvtuypcvn+v85XsVt23bdtVrxMfHk56e7nocOHCguNIVEREREZFSTtNKPeTyFNDq1avneb5mzZpu7fJisViwWCxFn5yIiIiIiJQ5Gjn0kICAAACOHj2a5/kjR464tRMRERERESlOKg6Lmbe3NwDZ2dlux+vXr4+fnx+bN2/m/PnzueJsNhsAjRo1Ku4URUREREREVBwWtypVqgDkuh/Q19eXhx9+mBMnTpCUlOR2LiUlhWXLllG3bl1atmx53XIVEREREZGyS/ccFrP69esTEhLCrFmzsFgs1K5dG8MwGDJkCOPGjWPVqlW88sorrFu3jnvuuYfU1FTmzJlD+fLl+fjjj6+6jYWIiIiIiEhRMpxOp9PTSZR2GzduZMSIEfzwww+cOXMGgP379xMeHs7w4cN5/fXXqV69OqdOncJqtRIZGcncuXNp06aNa3ppftjt9kvxxOJjlCumVyNlgXdh7nX1Nfe1Z/j5me5y34Aw07FhX501FeeTmvf9wvmRcy73VPJ8x/7fz5ACMwzTfWL214Qn+iyMkpavFC99PUhJVZiv3cIw+XVvlPM13aXhZ25hxsL8Hi4UZ465OMPcYFGWMxNbznzS09PzvY6JhqVMsNlsGIaRa1P7q7nnnnuw2WzY7XacTidOp5Pw8HAA1zYWs2bNIiMjg+PHjzNnzpxiylxERERERCRvKg5FRERERERExaGIiIiIiIioOCywhIQEoqKiAEhMTMQwDNcjNTUVgIyMDCZOnMhdd91FhQoVqFSpEvfddx9ffPGFBzMXERERERG5Oq1WWkCRkZGkpqYyY8YM2rRpQ2RkpOtcYGAgDoeDmJgYbDYbjRo1ol+/fmRmZrJkyRJiY2OZNGkSTzzxhOdegIiIiIiISB5UHBbQ5WJwxowZREZG5lqU5sUXX8RmszFy5EjXyCLAmTNniI6OZtiwYXTu3JmQkJBC5+JwOHA4HK7ndru90NcUEREREZGySdNKi1BOTg5Tpkzh5ptvdisMASpVqsS///1vMjIymD9/fpH0l5SUhNVqdT1CQ0OL5LoiIiIiIlL2aOSwCO3evZvTp08TEhJCYmJirvPHjx8HYNeuXUXSX3x8PEOHDnU9t9vtKhBFRERERMQUFYdF6NSpUwDs2LGDHTt2XLXduXPniqQ/i8WCxWJu808REREREZErqTgsQgEBAQB06dKFuXPnejgbERERERGR/NM9hyZ4e3sDkJ2d7Xb8tttuIyAggC1btpCZmemJ1ERERERERExRcWhClSpVADhw4IDbcR8fH/71r3/x22+/8eyzz+ZZIP78888cO3bsuuQpIiIiIiKSX5pWakL9+vUJCQlh1qxZWCwWateujWEYDBkyhMTERH744QfefvttlixZQuvWrQkODubgwYP89NNPbN++nfXr1xMcHOzplyEiIiIiIuKi4tAEb29v5s+fz4gRI0hOTubMmTMAPProo4SHh/PVV1/x0Ucf8cknnzBv3jwcDgfVq1fn9ttvZ9CgQfzlL3/x8CsQERERERFxZzidTqenkygtevfuzYwZM9i/fz/h4eHXvX+73Y7VaiWSWHyMcte9f5HC8A4KMh1reJufIZ9z5qypuNNx5v/IU3XVgWs3uoqs/x00FVeY9zcnLd1coDPHdJ/OrCzTsYbJVZy9KlU03afp94jCvVa5MRk+5v72rq+FG9QV+1ZfVyXsI7pRztdcnK/5z6xeQVVNxWUfPmq6T2dGhulYr/LlzcUFWk3FZeU4+PbQVNLT010LZ16zL1M9lWE2mw3DMEhISPB0KiIiIiIiIkVGxaGIiIiIiIioOBQREREREREVhwWSkJBAVFQUAImJiRiG4Xqkpqa62jmdTt5++23q16+PxWIhLCyMxMREcnLyvv9m0aJFtG3blsqVK+Pn50eDBg0YP358rn0URUREREREiotWKy2AyMhIUlNTmTFjBm3atCEyMtJ1LjAw0PX/zz33HKtWraJTp0507NiRhQsXkpCQQEZGBmPGjHG7Znx8PGPHjqVWrVp07twZq9XK6tWree6559i4cSNz5sy5Tq9ORERERETKMhWHBXC5GJwxYwaRkZFXXZTmhx9+4Mcff6RmzZoAjBw5knr16jFp0iRGjRqFr++l1Zy++eYbxo4dS8eOHZk3bx4VKlQALo08Dh48mPfee4958+bRpUuXPPtxOBw4HA7Xc7vdXkSvVEREREREyhpNKy0GI0eOdBWGANWqVSM2NpYzZ86we/du1/HJkycD8P7777sKQwDDMBg7diyGYZCcnHzVfpKSkrBara5HaGhoMbwaEREREREpCzRyWAyaNGmS61jt2rUBSEtLcx3bsGEDFSpUYNq0aXlex9/fn127dl21n/j4eIYOHep6brfbVSCKiIiIiIgpKg6LQV6bTPr834a4Vy4yc+rUKbKyskhMTLzqtc6dO3fVcxaLBYvJzZ5FRERERESupOLQgwICAjAMgxMnTng6FRERERERKeN0z2EBeXt7AxTJNhP33HMPJ0+eZM+ePYW+loiIiIiISGGoOCygKlWqAHDgwIFCX+vJJ58EoG/fvpw8eTLX+SNHjrBz585C9yMiIiIiInItmlZaQPXr1yckJIRZs2ZhsVioXbs2hmEwZMiQAl8rJiaGkSNHMnr0aOrWrUtMTAxhYWGcPHmSvXv3snr1al555RVuu+22YnglIiIiIiIi/5+KwwLy9vZm/vz5jBgxguTkZM6cOQPAo48+aup6L7/8Mq1bt+btt99m+fLlpKWlUbVqVSIiIkhISOCRRx4pyvRFbljZx4+bD/byNh1qeJuLrfzFDtN97nuugenYsIQjpuI88f76VA8y3WXWkaOmY50ZGabics6cNd2nd2gt07FZ+38zHVuiGIb5WKez6PK4DpxZWabivCpVMt1nzv99HikxzH49eOJroYR9/RVKIb5PnZnmfvbizDHdp9l/G6+w2qa7zN6bajo2508WmvzTuPPnTcVlOTMLHGM4nWXpK750s9vtWK1WIonFxyjn6XRErh8PFIeGn/mVgvcXqjjcZC4wpxD3SZew4tAsw9fXdKx3SA3TsSoO86GMfFRRcZgPZeRrwWM88H1q+Jgfq/KuVfPajfLgtJj/eV+Y4tD072KT/y5ZzkxszoWkp6fnuZtCXnTPoYiIiIiIiKg4FBERERERkTJQHM6bN482bdoQHByMn58fISEhtGvXjnnz5rnaTJs2jdjYWMLDw/Hz86NKlSp07NiRlStX5rqezWbDMAwSEhJYt24dUVFRVKpUiaCgIAYPHsyFCxcAWLJkCS1atKBChQpUr16d4cOHk3WVexAWLVpE27ZtqVy5Mn5+fjRo0IDx48cXyXYZIiIiIiIi+VGqi8MpU6bQtWtX9uzZw0MPPcTQoUOJiYnhyJEjLFiwwNXu8ccf5+jRo7Rr145nnnmGTp06sX79etq1a8eiRYvyvPbGjRtp27YtVquVgQMHctNNNzFlyhQGDBjA7Nmz6dq1K2FhYQwcOJDAwEBef/11Xn311VzXiY+PJy4ujt27d9O5c2cGDx6Mv78/zz33HD169Ci290ZERERERORKpXpBmiZNmvDzzz9z4MABgoOD3c6dPHmSqlWrArB//34iIiLczh8+fJimTZtSoUIFfv31V9dxm81GVFQUAAsXLiQ2NhaAzMxMmjZtyk8//UTVqlVZunQpzZo1A+DMmTPUrVuXrKwsjhw5QrlylxaL+eabb+jQoQMdO3Zk3rx5VKhQAQCn08ngwYN57733mDt3Ll26dMnz9TkcDhwOh+u53W4nNDRUC9JI2aMFaa5NC9JckxakKWZakOaatCBNPpSRrwWP0YI016QFaUq4cuXKuYqxK10uDIFchSFAzZo16dKlC3v27OG333L/4o6KinIVhpf76dq1K06nkwcffNBVGAJUqlSJTp06cerUKf73v/+5jk+ePBmA999/31UYAhiGwdixYzEMg+Tk5Ku+tqSkJKxWq+sRGhp61bYiIiIiIiJ/plTvc9ijRw+GDx9OgwYN6NmzJ1FRUbRq1SpX5fzf//6XpKQkVqxYwcGDB91G4wAOHTpEWFiY27FGjRrl6q9mzZrXPHfo0CFXMbphwwYqVKjAtGnT8szf39+fXbt2XfX1xcfHM3ToUNfzyyOHIiIiIiIiBVWqi8Nnn32WqlWrMmXKFCZMmMD48ePx8fHhgQce4I033iAiIoK9e/dy9913Y7fbiYqK4sEHHyQgIAAvLy9sNhurVq3KVSwCeQ7N+vzfsPifncvM/P+bUZ46dYqsrCwSExOv+hrO/clmmRaLBYvF/NQ2ERERERGRy0p1cWgYBn379qVv376cPHmS1atXk5yczOeff86ePXv48ccfeeONNzh9+jQzZ87k0UcfdYsfNGgQq1atKrb8AgICMAyDEydOFFsfIiIiIiIi+VHq7zm8rGrVqsTFxTF79myio6P55Zdf2Lt3L/v27QNwu38QLi0Ks3bt2mLN6Z577uHkyZPs2bOnWPsRERERERG5llJdHNpsNv64GGtmZianTp0CwM/Pz3Uv4Zo1a9zajR07lp9//rlY83vyyScBXCObf3TkyBF27txZrDmIiIiIiIhAKZ9WGhcXR0BAAM2bNycsLIzMzEy++eYbfvnlF9c+hIMGDeLjjz+mS5cudOvWjapVq7JhwwZ++OEHHnjgAZYsWVJs+cXExDBy5EhGjx5N3bp1iYmJISwsjJMnT7J3715Wr17NK6+8wm233VZsOYiIiIiIiEApLw6TkpJISUlh06ZNLF68mAoVKnDzzTczZcoU+vXrB0Djxo35+uuveemll5g/fz7e3t7ce++9rF27li+++KJYi0OAl19+mdatW/P222+zfPly0tLSqFq1KhERESQkJPDII48Ua/8iIiIiIiIAhvOP8y7LoN69ezNjxgz2799PeHj4NdvbbDaioqIYNWoUCQkJxZ5fftntdqxWK5HE4mPk3ttR5IZWiI3sC7PBu1HO3Ea4XhX8zffp52c69uiDdUzFVf1ok+k+vatWMRXn/JPVlq/FK8D8ZuDZJ0+binNmF+brqBCbONcINhWX9fv/rt3oajzwq9/s9xqAMyvz2o3yDPTQRxyTG1YbPuZ/d3vkPfLAhulSzArxu9i7YoVrN8pD9pkzpvs0vM3l6x1UzXSfWUeOmo71MrnLgDM7x1RcljOTlZlzSE9Pz3M3hbyU6nsOr2Sz2TAM44Yq5kRERERERG4UpXpaaXG5++672blzJ9Wqmf+rg4iIiIiIyI1ExaEJ5cuXp379+p5OQ0REREREpMiUiWmlCQkJREVFAZCYmIhhGK5Hamqqq53T6eTtt9+mfv36WCwWwsLCSExMJCfHfZ7v1aaohoeHEx4eztmzZ3nqqacICQnBYrHQsGFD5s6dm2duqampdO/enSpVqlCxYkXatGnDd999R0JCAoZhYLPZivKtEBERERERyVOZGDmMjIwkNTWVGTNm0KZNGyIjI13nAgMDXf//3HPPsWrVKjp16kTHjh1ZuHAhCQkJZGRkMGbMmHz1lZmZSYcOHTh9+jRdunTh/PnzzJo1i27dupGSkkKHDh1cbQ8ePMi9997L4cOHiYmJoXHjxuzevZv27dsTHR1dVC9fRERERETkmspMcQgwY8YMIiMjr7oozQ8//MCPP/5IzZo1ARg5ciT16tVj0qRJjBo1Cl/fa6+0dujQIZo1a4bNZnO179mzJ+3atWPixIluxeHzzz/P4cOHGTNmDC+88ILr+LRp01xbbfwZh8OBw+FwPbfb7deMERERERERyUuZmFaaXyNHjnQVhgDVqlUjNjaWM2fOsHv37nxf54033nArJNu2bUtYWBibN292HXM4HMyZM4fg4GCGDRvmFt+nTx9uvfXWa/aTlJSE1Wp1PUJDQ/Odo4iIiIiIyJVUHF6hSZMmuY7Vrl0bgLS0tHxdIzAwkIiIiDyvc+U1du/ejcPhoGnTplj+sOeJYRjce++91+wrPj6e9PR01+PAgQP5ylFEREREROSPysS00vzKa3NIH59Lb1F2PjdHtlqteR738fFxW9jm8hTQ4OC8Nz+uXr36NfuyWCy5CksREREREREzNHLoIZcL0WPHjuV5/ujRo9czHRERERERKePKTHHo7e0N5H8EsLjdeuutWCwWvv/+e7dFZeDSlhrr16/3UGYiIiIiIlIWlZnisEqVKgA3zH15FouFrl27cvToUd588023c5988gm7du3yTGIiIiIiIlImlZl7DuvXr09ISAizZs3CYrFQu3ZtDMNgyJAhHsspKSmJb7/9lueff55Vq1a59jn88ssviYmJISUlBS+vMlO/i4iIiIiIB5WZ4tDb25v58+czYsQIkpOTOXPmDACPPvqox3IKDQ1l/fr1jBgxgq+//ppVq1bRpEkTvv76a+bMmQPkvUiOiIiIiIhIUTOcTqfT00lIbq1atWL9+vWkp6dTsWLFfMXY7XasViuRxOJjlCt4p17eBY8ByLkx7uMUua7Mfr9Aob5nvCpVMhVnGIbpPvH3MxXmTLeb7tLZoK7pWK8DeS/0dS05aemm+3RmZJiO9baa/COgYX5mSc7//YG0oJxZWab7lGJm9nu8pH0MLMzPXk8ozGekEva5zCjne+1GeXHmXLtNETMKs9p/Ib5nchqa+91mZJr7N83KdrBy61jS09PzPeCkOYsedvjw4VzHPv30U9auXUu7du3yXRiKiIiIiIgURpmZVnqjatCgAY0bN+b222/H29ubbdu2YbPZqFSpEuPHj/d0eiIiIiIiUkaoOPSwQYMGsXjxYrZs2cK5c+cICgqiZ8+ejBw5kvr163s6PRERERERKSNUHHrYmDFjGDNmjKfTEBERERGRMk73HIqIiIiIiIhGDksyh8OBw+FwPbfbza8MKCIiIiIiZZtGDkuwpKQkrFar6xEaGurplEREREREpIRScViCxcfHk56e7nocOHDA0ymJiIiIiEgJpWmlJZjFYsFSmE08RURERERE/o9GDkVERERERETFoYiIiIiIiKg4vGHt27ePXbt2kZmZ6elURERERESkDFBxeINq27Ytt912GwcPHvR0KiIiIiIiUgZoQRpx8fItZyou52J2EWciUgI4czzSbc6ZM6bifGpUN91n9omT5gIN839/NH75r+lYyvubCvMqxAJf2VfsOVvg2LR0U3HeQUGm+zT8zb1HTpNff3IdOJ3m4gzj+vdZGDnmP3MYJr/HDV9f033mnDtvOrakcWZmmIrzDggo4kyuLacQP7O9ypc3Hettv2gq7nSjKqbisjPLwdaCxZS6kcPp06djGIbr0aNHD0+nVCAXL17EMAx+++03ACIiIjyckYiIiIiIlAWlrji8LDY2llGjRtG1a1fXsd69e7sVjnk9pk+f7mp/ZaGZlJSUZz9jx47NFQcQHh7udl2LxUJQUBB33303jz/+OGvWrMnzej4+PowaNYpRo0YRFhZW6PdBREREREQkP0rttNK4uDh69+6d57l+/fpRu3btPM81atQoz+Pjxo1j4MCBVKmS/2Fdb29vXnrpJQCysrI4ffo0P/30E1OnTuXdd9/lwQcfZMaMGVSuXNkV4+PjQ0JCAgA2m801gigiIiIiIlKcSm1x+Gf69+9P8+bN893+5ptvZt++fYwZM4YJEybkO+7KQu9Kv/32G/369WPx4sU89NBDrFixAi+vUjuIKyIiIiIiJYAqknzo3bs3devW5Z133uH3338v9PXCwsJYvHgxt912G6tWrWLu3LlFkKWIiIiIiIh5Kg7zwcfHhzFjxuBwOBg5cmSRXNPf359nn30WgNmzZ5u6hsPhwG63uz1ERERERETMKJPTSj/88ENSUlLyPPf888/j5+eX6/jf//53xo8fz6effsqwYcNo2LBhofOIjIwEYPPmzabik5KSSExMLHQeIiIiIiIiZbI4/Oijj6567umnn86zODQMg3HjxhEdHc3zzz/P0qVLC51HSEgIACdOnDAVHx8fz9ChQ13P7XY7oaGhhc5LRERERETKnjI5rXT9+vU4nc48H4GBgVeNi4qKIiYmhq+++opVq1Zdv4SvwmKxEBAQ4PYQERERERExo0wWh4UxduxYvLy8GD58eKGvdejQIQCCgoIKfS0REREREZHCUHFYQHfeeSePPPIImzZtYs6cOYW6ls1mA6BZs2ZFkJmIiIiIiIh5Kg5NGD16NBaLhRdffJGsrCxT17hw4YJrz8SHH364KNMTEREREREpMBWHJoSFhTF48GD27NnD9OnTCxz/+++/8+CDD/LLL78QFRVF586diz5JERERERGRAiiTq5X+2VYWzZs3JyYm5prXePHFF5k2bRr79u27apusrCwSEhIAyM7OJi0tjR9//JG1a9eSnZ1NbGws06dPxzAMU6/jj5xO56V+yQRnweO9nOb+VpDjzDQVJ1KyFeL71mniG7SwcjJMh2ab/h43//dHw5ljPjbH21xgIX6WmX+PzHMW4t/U6TQXq5/3pVEJ+1lWCIbJzzlGIV6mR75nnNnXv89CMPvzqDAK8+/iVZh8sx3mwjIvFirOWYDv1TJZHP7ZVhZPPfVUvorDqlWrMmLECF544YWrtsnOznbtQ+jr60tAQAAREREMHDiQnj170rJly4In/yfOnDkDwBpMbrNh7utOpGwqWZ+J4KinEyigwnyeOl9kWdzYzO2CJOKupP0sKwxzn8vNx0n+2D2dQAEV5uvhtMm4XYXok0s1gtVqzVdbw1mQUrIEmD59On369OHjjz+md+/enk6nUCIjI1m1alW+q/2cnBwOHTpEpUqV8hyNvLwP4oEDBwq07YXZuJLWp/ItfX0q39LXp/ItfX0q39LXZ0nLV+9R6ctX79ElTqeTM2fOEBISgpdX/kbOS+3IYZ8+fejTpw/du3dn1qxZnk4n3y5evIi/v7+pWC8vL2rXrn3Ndmb3RCzMXoolqc/CxCrfG7PPwsQq3xuzz8LEKt8bs8/CxCrfG7PPwsSWlT4LE6t8b8w+CxNbHH3md8TwslJXHDZq1IhRo0a5njdo0MCD2RScj4+PW/4iIiIiIiLXQ6ksDhs1auTpNEzz8fFxLWIjIiIiIiJyvWgrizLEYrEwatQoLBbLdYkraX0WJlb53ph9FiZW+d6YfRYmVvnemH0WJlb53ph9Fia2rPRZmFjle2P2WZhYT+Wbl1K3II2IiIiIiIgUnEYORURERERERMWhiIiIiIiIqDgUERERERERVByKiIiIiIgIKg5FRESKXO/evYmLi7vu/U6fPp3AwMBrtktISCiWbZ+K67oiInJ9qDgUERERERERFYciIiLFLTIykieffJLhw4dTpUoVatSoQUJCglsbwzCYMmUKf/3rX/H396dOnTrMnTvXdd5ms2EYBmlpaa5j27ZtwzAMUlNTsdls9OnTh/T0dAzDwDCMXH3ApdHFxMREtm/f7mo3ffp0ANLS0ujfvz9BQUEEBAQQHR3N9u3bATh+/Dg1atTg1VdfdV1r3bp1+Pr6snz58j+9roiIlAw+nk5ARESkLJgxYwZDhw5l48aNrF+/nt69e9OyZUvat2/vajNy5EjGjh3LW2+9xcyZM+nRowc//fQTt9122zWvf++99/Lmm2/y73//m927dwNQsWLFXO26d+/Ozz//TEpKCt9++y0AVqsVgL///e/4+/vz1VdfYbVamTp1Km3btuXXX38lKCiIadOmERcXR4cOHbj11lvp1asXTzzxBG3btuXChQtXva6IiJQMKg5FRESug4YNGzJq1CgA6tWrx+TJk1m+fLlbcfj3v/+d/v37AzB69Gi++eYbJk2axLvvvnvN6/v6+mK1WjEMgxo1aly1nb+/PxUrVsTHx8et3Zo1a9i0aRPHjh3DYrEAMH78eBYuXMjcuXP55z//yf3338+AAQN45JFHaNq0KRUqVCApKelPrysiIiWHikMREZHroGHDhm7Pa9asybFjx9yOtWjRItfzbdu2FXdqAGzfvp2zZ89StWpVt+MXLlxg3759rufjx4+nQYMGzJkzh++//95VSIqISMmn4lBEROQ6KFeunNtzwzDIycnJd7yX16VlApxOp+tYZmZm0SQHnD17lpo1a2Kz2XKdu3IF1H379nHo0CFycnJITU3lL3/5S5HlICIinqXiUERE5AaxYcMG/vGPf7g9b9y4MQBBQUEAHD58mMqVKwPkGlX09fUlOzv7mv3k1e6uu+7iyJEj+Pj4EB4enmdcRkYGjz76KN27d+fWW2+lf//+/PTTTwQHBxeofxERuTFptVIREZEbxJw5c5g2bRq//voro0aNYtOmTTzxxBMA1K1bl9DQUBISEtizZw9LlixhwoQJbvHh4eGcPXuW5cuXc+LECc6fP59nP+Hh4ezfv59t27Zx4sQJHA4H7dq1o0WLFsTFxfH111+TmprKunXrePHFF9myZQsAL774Iunp6bz99tuMGDGCW265hb59+/7pdUVEpORQcSgiInKDSExMZNasWTRs2JBPPvmE5ORkbr/9duDStNTk5GR27dpFw4YNGTduHK+88opb/L333sugQYPo3r07QUFBvPbaa3n206VLF2JiYoiKiiIoKIjk5GQMw2Dp0qW0bt2aPn36cMstt9CjRw9+++03qlevjs1m480332TmzJkEBATg5eXFzJkzWb16NVOmTLnqdUVEpOQwnFfevCAiIiIeYRgGCxYsIC4uztOpiIhIGaWRQxEREREREVFxKCIiIiIiIlqtVERE5IaguzxERMTTNHIoIiIiIiIiKg5FRERERERExaGIiIiIiIig4lBERERERERQcSgiIiIiIiKoOBQRERERERFUHIqIiIiIiAgqDkVERERERAQVhyIiIiIiIgL8P1C48hwT/D4hAAAAAElFTkSuQmCC",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"model.plot_attention(long_text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PToqG3GiIUPM"
},
"source": [
"The `translate` function works on batches, so if you have multiple texts to translate you can pass them all at once, which is much more efficient than translating them one at a time:"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:12.817770Z",
"iopub.status.busy": "2023-12-07T12:24:12.817511Z",
"iopub.status.idle": "2023-12-07T12:24:12.820930Z",
"shell.execute_reply": "2023-12-07T12:24:12.820205Z"
},
"id": "1-FLCjBVEMXL"
},
"outputs": [],
"source": [
"inputs = [\n",
" 'Hace mucho frio aqui.', # \"It's really cold here.\"\n",
" 'Esta es mi vida.', # \"This is my life.\"\n",
" 'Su cuarto es un desastre.' # \"His room is a mess\"\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:12.824040Z",
"iopub.status.busy": "2023-12-07T12:24:12.823764Z",
"iopub.status.idle": "2023-12-07T12:24:13.345913Z",
"shell.execute_reply": "2023-12-07T12:24:13.345274Z"
},
"id": "sT68i4jYEQ7q"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"its cold here . \n",
"this is my life . \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"her room is a mess . \n",
"\n",
"CPU times: user 505 ms, sys: 20 ms, total: 525 ms\n",
"Wall time: 518 ms\n"
]
}
],
"source": [
"%%time\n",
"for t in inputs:\n",
" print(model.translate([t])[0].numpy().decode())\n",
"\n",
"print()"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:13.349597Z",
"iopub.status.busy": "2023-12-07T12:24:13.349085Z",
"iopub.status.idle": "2023-12-07T12:24:14.338624Z",
"shell.execute_reply": "2023-12-07T12:24:14.337919Z"
},
"id": "hd2rgyHwVVrv"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"its cold here . \n",
"this is my life . \n",
"her room is a mess . \n",
"\n",
"CPU times: user 964 ms, sys: 9.95 ms, total: 974 ms\n",
"Wall time: 985 ms\n"
]
}
],
"source": [
"%%time\n",
"result = model.translate(inputs)\n",
"\n",
"print(result[0].numpy().decode())\n",
"print(result[1].numpy().decode())\n",
"print(result[2].numpy().decode())\n",
"print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uvhMqIw26Bwd"
},
"source": [
"So overall this text generation function mostly gets the job done, but so you've only used it here in python with eager execution. Let's try to export it next:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X4POAuUgLxLv"
},
"source": [
"### Export"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S-6cFyqeUPQm"
},
"source": [
"If you want to export this model you'll need to wrap the `translate` method in a tf.function
. That implementation will get the job done:"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:14.342139Z",
"iopub.status.busy": "2023-12-07T12:24:14.341897Z",
"iopub.status.idle": "2023-12-07T12:24:14.346466Z",
"shell.execute_reply": "2023-12-07T12:24:14.345818Z"
},
"id": "fNhGwQaVKIAy"
},
"outputs": [],
"source": [
"class Export(tf.Module):\n",
" def __init__(self, model):\n",
" self.model = model\n",
"\n",
" @tf.function(input_signature=[tf.TensorSpec(dtype=tf.string, shape=[None])])\n",
" def translate(self, inputs):\n",
" return self.model.translate(inputs)"
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:14.349374Z",
"iopub.status.busy": "2023-12-07T12:24:14.348916Z",
"iopub.status.idle": "2023-12-07T12:24:14.352239Z",
"shell.execute_reply": "2023-12-07T12:24:14.351614Z"
},
"id": "5Tjqs9FzNwW5"
},
"outputs": [],
"source": [
"export = Export(model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fkccvHDvXCa8"
},
"source": [
"Run the tf.function
once to compile it:"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:24:14.354869Z",
"iopub.status.busy": "2023-12-07T12:24:14.354649Z",
"iopub.status.idle": "2023-12-07T12:25:11.701084Z",
"shell.execute_reply": "2023-12-07T12:25:11.700336Z"
},
"id": "_NzrixLvVBjQ"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
"W0000 00:00:1701951901.224441 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.225287 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.226032 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.226745 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.227451 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.228078 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.228713 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.229364 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.229992 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.230627 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.231267 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.231944 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.232908 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.233545 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.234161 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.234858 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.235588 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.236245 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.236899 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.237535 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.238175 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.238802 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.239424 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.240045 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.240671 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.241300 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.241917 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.242583 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.243249 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.243870 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.244527 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.245280 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.245977 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.246629 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.247283 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.247925 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.248557 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.249210 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.249830 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.250500 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.251124 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.251756 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.252394 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.253056 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.254154 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.254875 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.255603 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.256252 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.256880 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701951901.257556 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 56.8 s, sys: 1.37 s, total: 58.2 s\n",
"Wall time: 57.3 s\n"
]
}
],
"source": [
"%%time\n",
"_ = export.translate(tf.constant(inputs))"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:25:11.704503Z",
"iopub.status.busy": "2023-12-07T12:25:11.704000Z",
"iopub.status.idle": "2023-12-07T12:25:11.841253Z",
"shell.execute_reply": "2023-12-07T12:25:11.840595Z"
},
"id": "USJdu00tVFbd"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"its cold here . \n",
"this is my life . \n",
"her room is a mess . \n",
"\n",
"CPU times: user 132 ms, sys: 18.2 ms, total: 150 ms\n",
"Wall time: 132 ms\n"
]
}
],
"source": [
"%%time\n",
"result = export.translate(tf.constant(inputs))\n",
"\n",
"print(result[0].numpy().decode())\n",
"print(result[1].numpy().decode())\n",
"print(result[2].numpy().decode())\n",
"print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NP2dNtEXJPEL"
},
"source": [
"Now that the function has been traced it can be exported using saved_model.save
:"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:25:11.844840Z",
"iopub.status.busy": "2023-12-07T12:25:11.844301Z",
"iopub.status.idle": "2023-12-07T12:26:22.908655Z",
"shell.execute_reply": "2023-12-07T12:26:22.907946Z"
},
"id": "OyvxT5V0_X5B"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Assets written to: translator/assets\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:tensorflow:Assets written to: translator/assets\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 1min 10s, sys: 948 ms, total: 1min 11s\n",
"Wall time: 1min 11s\n"
]
}
],
"source": [
"%%time\n",
"tf.saved_model.save(export, 'translator',\n",
" signatures={'serving_default': export.translate})"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:26:22.914045Z",
"iopub.status.busy": "2023-12-07T12:26:22.913795Z",
"iopub.status.idle": "2023-12-07T12:27:11.535146Z",
"shell.execute_reply": "2023-12-07T12:27:11.534384Z"
},
"id": "-I0j3i3ekOba"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:26.948623: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:27.620729: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:27.719407: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:27.730802: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:28.840628: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:29.113641: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:29.297780: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:29.390506: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:29.401262: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:29.429236: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:29.512176: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:29.523005: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:29.612756: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:29.889963: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:29.904132: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:30.321676: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:30.333840: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:30.952562: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:31.587636: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:31.747954: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:31.758419: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:31.828059: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:31.841421: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:31.860615: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:31.871589: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:32.015126: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:32.036575: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:32.415935: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:32.427836: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:32.973277: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:32.987440: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:32.999725: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:33.010430: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:33.073042: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:33.083551: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:33.119754: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:33.449532: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:33.460910: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:33.697277: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:33.829566: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:33.840227: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:36.439691: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.451798: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.576061: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.586847: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.607916: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.618278: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:36.703713: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.725119: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.776850: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:36.945446: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:36.956754: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:37.578512: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:37.589238: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:37.638069: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:37.648438: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:37.660114: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:37.670468: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:37.811462: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:37.912472: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:37.923672: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.014409: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:38.026320: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.082647: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.094670: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:38.308973: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.321667: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:38.830232: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.841017: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.871593: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.882028: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.896500: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.907346: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:38.924352: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.010291: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:39.096337: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.106893: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.139453: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.150113: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:39.555243: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.566061: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.577701: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.587860: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.604954: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.614974: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.626945: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.665892: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:39.677070: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:39.827448: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:40.047370: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.058536: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.076495: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.087785: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.105670: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.116125: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:40.437676: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.449112: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.614461: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:40.850527: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.861031: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:40.943945: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:41.359792: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:41.370255: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:41.864374: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:41.932957: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:41.944267: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:41.963262: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:41.974456: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:42.859014: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:42.870901: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:42.929756: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:42.941754: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:42.952280: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:42.969764: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:43.051864: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:43.062528: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:43.177318: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:43.187645: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:43.246919: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:43.257070: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:44.539314: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:44.550434: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:44.567303: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:44.658590: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:44.669101: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:44.887137: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:44.897819: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:45.000661: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:45.010826: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:26:45.182639: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:45.193465: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:45.210968: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:45.365541: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:26:45.376302: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"W0000 00:00:1701952016.990642 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.991370 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.992043 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.992693 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.993351 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.994023 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.994770 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.995509 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.996148 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.996788 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.997487 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.998166 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.999149 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952016.999851 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.000524 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.001202 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.001895 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.002573 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.003268 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.003988 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.004741 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.005482 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.006155 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.006843 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.007525 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.008194 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.008949 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.009698 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.010417 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.011120 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.011795 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.012473 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.013160 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.013834 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.014611 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.015367 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.016038 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.016720 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.017422 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.018103 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.018792 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.019504 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.020177 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.020890 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.022377 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.023092 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.023808 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.024598 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.025454 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n",
"W0000 00:00:1701952017.026125 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 48 s, sys: 1.96 s, total: 49.9 s\n",
"Wall time: 48.6 s\n"
]
}
],
"source": [
"%%time\n",
"reloaded = tf.saved_model.load('translator')\n",
"_ = reloaded.translate(tf.constant(inputs)) #warmup"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:11.538688Z",
"iopub.status.busy": "2023-12-07T12:27:11.538405Z",
"iopub.status.idle": "2023-12-07T12:27:11.681515Z",
"shell.execute_reply": "2023-12-07T12:27:11.680681Z"
},
"id": "GXZF__FZXJCm"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"its cold here . \n",
"this is my life . \n",
"her room is a mess . \n",
"\n",
"CPU times: user 131 ms, sys: 29.1 ms, total: 160 ms\n",
"Wall time: 138 ms\n"
]
}
],
"source": [
"%%time\n",
"result = reloaded.translate(tf.constant(inputs))\n",
"\n",
"print(result[0].numpy().decode())\n",
"print(result[1].numpy().decode())\n",
"print(result[2].numpy().decode())\n",
"print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Pgg3P757O5rw"
},
"source": [
"#### [Optional] Use a dynamic loop"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3230LfyRIJQV"
},
"source": [
"It's worth noting that this initial implementation is not optimal. It uses a python loop:\n",
"\n",
"```\n",
"for _ in range(max_length):\n",
" ...\n",
" if tf.executing_eagerly() and tf.reduce_all(done):\n",
" break\n",
"```\n",
"\n",
"The python loop is relatively simple but when tf.function
converts this to a graph, it **statically unrolls** that loop. Unrolling the loop has two disadvantages:\n",
"\n",
"1. It makes `max_length` copies of the loop body. So the generated graphs take longer to build, save and load.\n",
"1. You have to choose a fixed value for the `max_length`. \n",
"1. You can't `break` from a statically unrolled loop. The tf.function
\n",
" version will run the full `max_length` iterations on every call.\n",
" That's why the `break` only works with eager execution. This is\n",
" still marginally faster than eager execution, but not as fast as it could be."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zPRJp4TRJx_n"
},
"source": [
"To fix these shortcomings, the `translate_dynamic` method, below, uses a tensorflow loop:\n",
"\n",
"```\n",
"for t in tf.range(max_length):\n",
" ...\n",
" if tf.reduce_all(done):\n",
" break\n",
"```\n",
"\n",
"It looks like a python loop, but when you use a tensor as the input to a `for` loop (or the condition of a `while` loop) tf.function
converts it to a dynamic loop using operations like tf.while_loop
. \n",
"\n",
"There's no need for a `max_length` here it's just in case the model gets stuck generating a loop like: `the united states of the united states of the united states...`.\n",
"\n",
"On the down side, to accumulate tokens from this dynamic loop you can't just append them to a python `list`, you need to use a tf.TensorArray
:\n",
"\n",
"```\n",
"tokens = tf.TensorArray(tf.int64, size=1, dynamic_size=True)\n",
"...\n",
"for t in tf.range(max_length):\n",
" ...\n",
" tokens = tokens.write(t, next_token) # next_token shape is (batch, 1)\n",
" ...\n",
" tokens = tokens.stack()\n",
" tokens = einops.rearrange(tokens, 't batch 1 -> batch t')\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rTmISp4SRo5U"
},
"source": [
"This version of the code can be quite a bit more efficient:"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:11.685585Z",
"iopub.status.busy": "2023-12-07T12:27:11.685301Z",
"iopub.status.idle": "2023-12-07T12:27:11.692588Z",
"shell.execute_reply": "2023-12-07T12:27:11.691904Z"
},
"id": "EbQpyYs13jF_"
},
"outputs": [],
"source": [
"#@title\n",
"@Translator.add_method\n",
"def translate(self,\n",
" texts,\n",
" *,\n",
" max_length=500,\n",
" temperature=tf.constant(0.0)):\n",
" shape_checker = ShapeChecker()\n",
" context = self.encoder.convert_input(texts)\n",
" batch_size = tf.shape(context)[0]\n",
" shape_checker(context, 'batch s units')\n",
"\n",
" next_token, done, state = self.decoder.get_initial_state(context)\n",
"\n",
" # initialize the accumulator\n",
" tokens = tf.TensorArray(tf.int64, size=1, dynamic_size=True)\n",
"\n",
" for t in tf.range(max_length):\n",
" # Generate the next token\n",
" next_token, done, state = self.decoder.get_next_token(\n",
" context, next_token, done, state, temperature)\n",
" shape_checker(next_token, 'batch t1')\n",
"\n",
" # Collect the generated tokens\n",
" tokens = tokens.write(t, next_token)\n",
"\n",
" # if all the sequences are done, break\n",
" if tf.reduce_all(done):\n",
" break\n",
"\n",
" # Convert the list of generated token ids to a list of strings.\n",
" tokens = tokens.stack()\n",
" shape_checker(tokens, 't batch t1')\n",
" tokens = einops.rearrange(tokens, 't batch 1 -> batch t')\n",
" shape_checker(tokens, 'batch t')\n",
"\n",
" text = self.decoder.tokens_to_text(tokens)\n",
" shape_checker(text, 'batch')\n",
"\n",
" return text"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AJ_NznOgZTxC"
},
"source": [
"With eager execution this implementation performs on par with the original:"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:11.695681Z",
"iopub.status.busy": "2023-12-07T12:27:11.695417Z",
"iopub.status.idle": "2023-12-07T12:27:11.922099Z",
"shell.execute_reply": "2023-12-07T12:27:11.921433Z"
},
"id": "JRh66y-YYeBw"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"its cold here . \n",
"this is my life . \n",
"her room is a mess . \n",
"\n",
"CPU times: user 217 ms, sys: 9.29 ms, total: 226 ms\n",
"Wall time: 222 ms\n"
]
}
],
"source": [
"%%time\n",
"result = model.translate(inputs)\n",
"\n",
"print(result[0].numpy().decode())\n",
"print(result[1].numpy().decode())\n",
"print(result[2].numpy().decode())\n",
"print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l6B8W4_MZdX0"
},
"source": [
"But when you wrap it in a tf.function
you'll notice two differences."
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:11.925681Z",
"iopub.status.busy": "2023-12-07T12:27:11.925409Z",
"iopub.status.idle": "2023-12-07T12:27:11.930140Z",
"shell.execute_reply": "2023-12-07T12:27:11.929523Z"
},
"id": "EQlrhWWrUhgT"
},
"outputs": [],
"source": [
"class Export(tf.Module):\n",
" def __init__(self, model):\n",
" self.model = model\n",
"\n",
" @tf.function(input_signature=[tf.TensorSpec(dtype=tf.string, shape=[None])])\n",
" def translate(self, inputs):\n",
" return self.model.translate(inputs)"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:11.933550Z",
"iopub.status.busy": "2023-12-07T12:27:11.932935Z",
"iopub.status.idle": "2023-12-07T12:27:12.133691Z",
"shell.execute_reply": "2023-12-07T12:27:12.132881Z"
},
"id": "pH8yyGHvUmti"
},
"outputs": [],
"source": [
"export = Export(model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZnOJvIsvUwBL"
},
"source": [
"First, it's much quicker to trace, since it only creates one copy of the loop body:"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:12.137716Z",
"iopub.status.busy": "2023-12-07T12:27:12.137443Z",
"iopub.status.idle": "2023-12-07T12:27:16.114826Z",
"shell.execute_reply": "2023-12-07T12:27:16.114093Z"
},
"id": "_CaEbHkwEa1S"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"W0000 00:00:1701952035.487670 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 4.02 s, sys: 39.5 ms, total: 4.06 s\n",
"Wall time: 3.97 s\n"
]
}
],
"source": [
"%%time\n",
"_ = export.translate(inputs)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2ABEwtKIZ6eE"
},
"source": [
"The tf.function
is much faster than running with eager execution, and on small inputs it's often several times faster than the unrolled version, because it can break out of the loop."
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:16.118497Z",
"iopub.status.busy": "2023-12-07T12:27:16.118229Z",
"iopub.status.idle": "2023-12-07T12:27:16.156412Z",
"shell.execute_reply": "2023-12-07T12:27:16.155680Z"
},
"id": "d5VdCLxPYrpz"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"its cold here . \n",
"this is my life . \n",
"her room is a mess . \n",
"\n",
"CPU times: user 45.1 ms, sys: 4.86 ms, total: 50 ms\n",
"Wall time: 33.6 ms\n"
]
}
],
"source": [
"%%time\n",
"result = export.translate(inputs)\n",
"\n",
"print(result[0].numpy().decode())\n",
"print(result[1].numpy().decode())\n",
"print(result[2].numpy().decode())\n",
"print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3DDmofICJdx0"
},
"source": [
"So save this version as well:"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:16.159548Z",
"iopub.status.busy": "2023-12-07T12:27:16.159293Z",
"iopub.status.idle": "2023-12-07T12:27:42.290857Z",
"shell.execute_reply": "2023-12-07T12:27:42.290149Z"
},
"id": "eCg7kRq6FVl3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Assets written to: dynamic_translator/assets\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:tensorflow:Assets written to: dynamic_translator/assets\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 26.1 s, sys: 107 ms, total: 26.2 s\n",
"Wall time: 26.1 s\n"
]
}
],
"source": [
"%%time\n",
"tf.saved_model.save(export, 'dynamic_translator',\n",
" signatures={'serving_default': export.translate})"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:42.294264Z",
"iopub.status.busy": "2023-12-07T12:27:42.294009Z",
"iopub.status.idle": "2023-12-07T12:27:53.096292Z",
"shell.execute_reply": "2023-12-07T12:27:53.095556Z"
},
"id": "zrpzxL2vFVl3"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:43.648070: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:44.408333: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:44.419266: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:45.101724: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.144001: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.154833: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.220104: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.230529: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:45.530241: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.663150: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.673852: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.722373: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.732531: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:45.750944: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:45.761396: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:46.418836: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:46.671461: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:46.913282: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.073802: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.084615: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.105871: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.116026: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:47.235939: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.246496: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:47.565299: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.576999: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.595454: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:47.874784: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:47.885903: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:48.164355: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 42 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:48.765149: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:48.776032: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:48.788116: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:48.799055: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-12-07 12:27:48.998277: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond/while' has 13 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n",
"2023-12-07 12:27:49.009647: W tensorflow/core/common_runtime/graph_constructor.cc:840] Node 'cond' has 4 outputs but the _output_shapes attribute specifies shapes for 40 outputs. Output shapes may be inaccurate.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"W0000 00:00:1701952071.866548 16852 op_level_cost_estimator.cc:699] Error in PredictCost() for the op: op: \"Softmax\" attr { key: \"T\" value { type: DT_FLOAT } } inputs { dtype: DT_FLOAT shape { unknown_rank: true } } device { type: \"GPU\" vendor: \"NVIDIA\" model: \"Tesla T4\" frequency: 1590 num_cores: 40 environment { key: \"architecture\" value: \"7.5\" } environment { key: \"cuda\" value: \"12020\" } environment { key: \"cudnn\" value: \"8904\" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 4194304 shared_memory_size_per_multiprocessor: 65536 memory_size: 14330101760 bandwidth: 320064000 } outputs { dtype: DT_FLOAT shape { unknown_rank: true } }\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 10.6 s, sys: 271 ms, total: 10.9 s\n",
"Wall time: 10.8 s\n"
]
}
],
"source": [
"%%time\n",
"reloaded = tf.saved_model.load('dynamic_translator')\n",
"_ = reloaded.translate(tf.constant(inputs)) #warmup"
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {
"execution": {
"iopub.execute_input": "2023-12-07T12:27:53.100102Z",
"iopub.status.busy": "2023-12-07T12:27:53.099298Z",
"iopub.status.idle": "2023-12-07T12:27:53.130152Z",
"shell.execute_reply": "2023-12-07T12:27:53.129471Z"
},
"id": "5TjSwrCEFVl3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"its cold here . \n",
"this is my life . \n",
"her room is a mess . \n",
"\n",
"CPU times: user 34 ms, sys: 4.46 ms, total: 38.4 ms\n",
"Wall time: 25.7 ms\n"
]
}
],
"source": [
"%%time\n",
"result = reloaded.translate(tf.constant(inputs))\n",
"\n",
"print(result[0].numpy().decode())\n",
"print(result[1].numpy().decode())\n",
"print(result[2].numpy().decode())\n",
"print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RTe5P5ioMJwN"
},
"source": [
"## Next steps\n",
"\n",
"* [Download a different dataset](http://www.manythings.org/anki/) to experiment with translations, for example, English to German, or English to French.\n",
"* Experiment with training on a larger dataset, or using more epochs.\n",
"* Try the [transformer tutorial](transformer.ipynb) which implements a similar translation task but uses transformer layers instead of RNNs. This version also uses a text.BertTokenizer
to implement word-piece tokenization.\n",
"* Visit the [`tensorflow_addons.seq2seq` tutorial](https://www.tensorflow.org/addons/tutorials/networks_seq2seq_nmt), which demonstrates a higher-level functionality for implementing this sort of sequence-to-sequence model, such as seq2seq.BeamSearchDecoder
."
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [],
"last_runtime": {
"build_target": "//learning/deepmind/public/tools/ml_python:ml_notebook",
"kind": "private"
},
"name": "nmt_with_attention.ipynb",
"private_outputs": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.18"
}
},
"nbformat": 4,
"nbformat_minor": 0
}