diff --git a/docs_nnx/examples/machine_translation.ipynb b/docs_nnx/examples/machine_translation.ipynb new file mode 100644 index 000000000..f89fe710e --- /dev/null +++ b/docs_nnx/examples/machine_translation.ipynb @@ -0,0 +1,1207 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "ee3e1116-f6cd-497e-b617-1d89d5d1f744", + "metadata": {}, + "source": [ + "# Machine Translation with encoder-decoder transformer model" + ] + }, + { + "cell_type": "markdown", + "id": "50f0bd58-dcc6-41f4-9dc4-3a08c8ef751b", + "metadata": {}, + "source": [ + "This tutorial is adapted from [Keras' documentation on English-to-Spanish translation with a sequence-to-sequence Transformer](https://keras.io/examples/nlp/neural_machine_translation_with_transformer/), which is itself an adaptation from the book [Deep Learning with Python, Second Edition by François Chollet](https://www.manning.com/books/deep-learning-with-python-second-edition)\n", + "\n", + "We step through an encoder-decoder transformer in JAX and train a model for English->Spanish translation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1OUXyjbIf6Fk", + "metadata": {}, + "outputs": [], + "source": [ + "# !pip install tiktoken grain flax optax" + ] + }, + { + "cell_type": "markdown", + "id": "d5597103", + "metadata": {}, + "source": [ + "Standard library and data handling imports:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2296aebd", + "metadata": {}, + "outputs": [], + "source": [ + "import pathlib\n", + "import random\n", + "import string\n", + "import re\n", + "import numpy as np" + ] + }, + { + "cell_type": "markdown", + "id": "517f7cf9", + "metadata": {}, + "source": [ + "JAX, Flax, and training framework imports:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "825de378", + "metadata": {}, + "outputs": [], + "source": [ + "import jax.numpy as jnp\n", + "import optax\n", + "\n", + "from flax import nnx" + ] + }, + { + "cell_type": "markdown", + "id": "42773685", + "metadata": {}, + "source": [ + "Tokenizer (`tiktoken`), data loader (`grain`), and progress bar (`tqdm`):" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "dd506ffa-3b91-44f1-92d1-a08ed933e78e", + "metadata": {}, + "outputs": [], + "source": [ + "import tiktoken\n", + "import grain.python as grain\n", + "import tqdm" + ] + }, + { + "cell_type": "markdown", + "id": "e1f324b0-140a-48fa-9fcb-d6308f098343", + "metadata": {}, + "source": [ + "## Pull down data to temp and extract into memory\n", + "\n", + "There are lots of ways to get this done, but for simplicity and clear visibility into what's happening this is downloaded to a temporary directory, extracted there, and read into a python object with processing." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "102943a5-8724-48e0-8d6a-f56069f03426", + "metadata": {}, + "outputs": [], + "source": [ + "import requests\n", + "import zipfile\n", + "import tempfile\n", + "\n", + "url = \"http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip\"\n", + "\n", + "with tempfile.TemporaryDirectory() as temp_dir:\n", + " temp_path = pathlib.Path(temp_dir)\n", + " zip_file_path = temp_path / \"spa-eng.zip\"\n", + "\n", + " response = requests.get(url)\n", + " zip_file_path.write_bytes(response.content)\n", + "\n", + " with zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n", + " zip_ref.extractall(temp_path)\n", + "\n", + " text_file = temp_path / \"spa-eng\" / \"spa.txt\"\n", + "\n", + " with open(text_file) as f:\n", + " lines = f.read().split(\"\\n\")[:-1]\n", + " text_pairs = []\n", + " for line in lines:\n", + " eng, spa = line.split(\"\\t\")\n", + " spa = \"[start] \" + spa + \" [end]\"\n", + " text_pairs.append((eng, spa))" + ] + }, + { + "cell_type": "markdown", + "id": "9524904b-fa17-493f-bcfa-335963cb7c45", + "metadata": {}, + "source": [ + "## Build train/validate/test pair sets\n", + "We'll stay close to the original tutorial so it's clear how to follow what's the same vs what's different; one early difference is the choice to go with an off-the-shelf encoder/tokenizer in tiktoken. Specifically \"cl100k_base\" - it has a wide range of language understanding and it's fast." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "bee9f1b0-5f74-47dc-a7e1-a4ea3be1ef7f", + "metadata": { + "outputId": "cf7fc8d5-029d-48d6-d739-95c53bdc9b38" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "118964 total pairs\n", + "83276 training pairs\n", + "17844 validation pairs\n", + "17844 test pairs\n" + ] + } + ], + "source": [ + "random.shuffle(text_pairs)\n", + "num_val_samples = int(0.15 * len(text_pairs))\n", + "num_train_samples = len(text_pairs) - 2 * num_val_samples\n", + "train_pairs = text_pairs[:num_train_samples]\n", + "val_pairs = text_pairs[num_train_samples : num_train_samples + num_val_samples]\n", + "test_pairs = text_pairs[num_train_samples + num_val_samples :]\n", + "\n", + "print(f\"{len(text_pairs)} total pairs\")\n", + "print(f\"{len(train_pairs)} training pairs\")\n", + "print(f\"{len(val_pairs)} validation pairs\")\n", + "print(f\"{len(test_pairs)} test pairs\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "fd1f61fe-e4b7-479d-917e-f609ebe482e9", + "metadata": {}, + "outputs": [], + "source": [ + "tokenizer = tiktoken.get_encoding(\"cl100k_base\")" + ] + }, + { + "cell_type": "markdown", + "id": "a714c4ea-9ff6-4dab-ae9c-1a884d4857e7", + "metadata": {}, + "source": [ + "We strip out punctuation to keep things simple and in line with the original tutorial - the `[` `]` are kept in so that our `[start]` and `[end]` formatting is preserved. We also record the vocabulary size from the tokenizer and fix the maximum sequence length for all inputs." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "07e054d3-a20c-4aed-8f8a-fb5158df8e5b", + "metadata": {}, + "outputs": [], + "source": [ + "strip_chars = string.punctuation + \"¿\"\n", + "strip_chars = strip_chars.replace(\"[\", \"\")\n", + "strip_chars = strip_chars.replace(\"]\", \"\")\n", + "\n", + "vocab_size = tokenizer.n_vocab\n", + "sequence_length = 20" + ] + }, + { + "cell_type": "markdown", + "id": "86ad4b78", + "metadata": {}, + "source": [ + "`custom_standardization` lowercases a string and removes the punctuation characters defined above:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "e2b3e5b3-8466-4c81-99da-0559c88b25ef", + "metadata": {}, + "outputs": [], + "source": [ + "def custom_standardization(input_string):\n", + " lowercase = input_string.lower()\n", + " return re.sub(f\"[{re.escape(strip_chars)}]\", \"\", lowercase)" + ] + }, + { + "cell_type": "markdown", + "id": "67f5d6ac", + "metadata": {}, + "source": [ + "`tokenize_and_pad` encodes a string into token IDs, truncates to `max_length` if needed, and zero-pads shorter sequences so every example is the same fixed length:" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "5bdc0673-9723-45b5-8a42-2152295df69b", + "metadata": {}, + "outputs": [], + "source": [ + "def tokenize_and_pad(text, tokenizer, max_length):\n", + " tokens = tokenizer.encode(text)[:max_length]\n", + " padded = tokens + [0] * (max_length - len(tokens)) if len(tokens) < max_length else tokens ##assumes list-like - (https://github.com/openai/tiktoken/blob/main/tiktoken/core.py#L81 current tiktoken out)\n", + " return padded" + ] + }, + { + "cell_type": "markdown", + "id": "28ed0a55", + "metadata": {}, + "source": [ + "`format_dataset` standardizes and tokenizes both the English and Spanish strings, then returns three arrays:\n", + "- `encoder_inputs` — the full tokenized English sentence.\n", + "- `decoder_inputs` — the Spanish sentence shifted right by one (all tokens except the last), used as the decoder prompt at each step.\n", + "- `target_output` — the Spanish sentence shifted left by one (all tokens except the first), used as the prediction target." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "235b1221-e72d-4793-addd-7bb870bd8e75", + "metadata": {}, + "outputs": [], + "source": [ + "def format_dataset(eng, spa, tokenizer, sequence_length):\n", + " eng = custom_standardization(eng)\n", + " spa = custom_standardization(spa)\n", + " eng = tokenize_and_pad(eng, tokenizer, sequence_length)\n", + " spa = tokenize_and_pad(spa, tokenizer, sequence_length)\n", + " return {\n", + " \"encoder_inputs\": eng,\n", + " \"decoder_inputs\": spa[:-1],\n", + " \"target_output\": spa[1:],\n", + " }" + ] + }, + { + "cell_type": "markdown", + "id": "f35e5045", + "metadata": {}, + "source": [ + "Apply the preprocessing to every split to produce the final in-memory datasets:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "ca013d07-1504-42cc-906f-2fcacc757008", + "metadata": {}, + "outputs": [], + "source": [ + "train_data = [format_dataset(eng, spa, tokenizer, sequence_length) for eng, spa in train_pairs]\n", + "val_data = [format_dataset(eng, spa, tokenizer, sequence_length) for eng, spa in val_pairs]\n", + "test_data = [format_dataset(eng, spa, tokenizer, sequence_length) for eng, spa in test_pairs]" + ] + }, + { + "cell_type": "markdown", + "id": "90bbae98-48dd-4ae4-99bb-92336d7c0a1c", + "metadata": {}, + "source": [ + "At this point we've extracted the data, applied formatting, and tokenized the phrases with padding. The data is kept in train/validate/test sets that each have dictionary entries, which look like the following:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "dcbfa780-553f-41f6-8b3e-55955db78b2a", + "metadata": { + "outputId": "017a7188-ff72-4f53-b725-be16f48cdd9f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'encoder_inputs': [1220, 539, 439, 4228, 439, 1274, 1781, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'decoder_inputs': [29563, 60, 912, 1560, 14531, 60240, 8112, 1208, 44142, 9115, 58925, 510, 408, 60, 0, 0, 0, 0, 0], 'target_output': [60, 912, 1560, 14531, 60240, 8112, 1208, 44142, 9115, 58925, 510, 408, 60, 0, 0, 0, 0, 0, 0]}\n" + ] + } + ], + "source": [ + "## data selection example\n", + "print(train_data[135])" + ] + }, + { + "cell_type": "markdown", + "id": "24c6271b-e359-4aba-a583-f18c40eddba9", + "metadata": {}, + "source": [ + "The output should look something like\n", + "\n", + "{'encoder_inputs': [9514, 265, 3339, 264, 2466, 16930, 1618, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'decoder_inputs': [29563, 60, 1826, 7206, 71086, 37116, 653, 16109, 1493, 54189, 510, 408, 60, 0, 0, 0, 0, 0, 0], 'target_output': [60, 1826, 7206, 71086, 37116, 653, 16109, 1493, 54189, 510, 408, 60, 0, 0, 0, 0, 0, 0, 0]}" + ] + }, + { + "cell_type": "markdown", + "id": "7a906a05-bd17-4a47-afe0-4422d2ea0f50", + "metadata": {}, + "source": [ + "## Define Transformer components: Encoder, Decoder, Positional Embed\n", + "\n", + "In many ways this is very similar to the original source, with `ops` changing to `jnp` and `keras` or `layers` becoming `nnx`. Certain module-specific arguments come and go, like the rngs attached to most things in the updated version, and `decode=False` in the `MultiHeadAttention` call.\n", + "\n", + "`TransformerEncoder` implements a standard encoder block: self-attention over the input sequence followed by a two-layer feed-forward projection, with a residual connection and layer norm after each sub-layer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1bd68b40", + "metadata": {}, + "outputs": [], + "source": [ + "class TransformerEncoder(nnx.Module):\n", + " def __init__(self, embed_dim: int, dense_dim: int, num_heads: int, rngs: nnx.Rngs, **kwargs):\n", + " self.embed_dim = embed_dim\n", + " self.dense_dim = dense_dim\n", + " self.num_heads = num_heads\n", + "\n", + " self.attention = nnx.MultiHeadAttention(num_heads=num_heads,\n", + " in_features=embed_dim,\n", + " decode=False,\n", + " rngs=rngs)\n", + " self.dense_proj = nnx.Sequential(\n", + " nnx.Linear(embed_dim, dense_dim, rngs=rngs),\n", + " nnx.relu,\n", + " nnx.Linear(dense_dim, embed_dim, rngs=rngs),\n", + " )\n", + "\n", + " self.layernorm_1 = nnx.LayerNorm(embed_dim, rngs=rngs)\n", + " self.layernorm_2 = nnx.LayerNorm(embed_dim, rngs=rngs)\n", + "\n", + " def __call__(self, inputs):\n", + " attention_output = self.attention(\n", + " inputs_q = inputs, inputs_k = inputs, inputs_v = inputs, decode = False\n", + " )\n", + " proj_input = self.layernorm_1(inputs + attention_output)\n", + " proj_output = self.dense_proj(proj_input)\n", + " return self.layernorm_2(proj_input + proj_output)" + ] + }, + { + "cell_type": "markdown", + "id": "2a186043", + "metadata": {}, + "source": [ + "`PositionalEmbedding` combines two learned embedding tables: one maps token IDs to vectors, the other maps position indices (0, 1, 2, …) to vectors. Their sum gives each token both a semantic and a positional representation. `compute_mask` returns a boolean array that marks non-padding tokens (anything that is not token ID 0)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0a66aa6e", + "metadata": {}, + "outputs": [], + "source": [ + "class PositionalEmbedding(nnx.Module):\n", + " def __init__(self, sequence_length: int, vocab_size: int, embed_dim: int, rngs: nnx.Rngs, **kwargs):\n", + " self.token_embeddings = nnx.Embed(num_embeddings=vocab_size, features=embed_dim, rngs=rngs)\n", + " self.position_embeddings = nnx.Embed(num_embeddings=sequence_length, features=embed_dim, rngs=rngs)\n", + " self.sequence_length = sequence_length\n", + " self.vocab_size = vocab_size\n", + " self.embed_dim = embed_dim\n", + "\n", + " def __call__(self, inputs):\n", + " length = inputs.shape[1]\n", + " positions = jnp.arange(0, length)[None, :]\n", + " embedded_tokens = self.token_embeddings(inputs)\n", + " embedded_positions = self.position_embeddings(positions)\n", + " return embedded_tokens + embedded_positions\n", + "\n", + " def compute_mask(self, inputs, mask=None):\n", + " if mask is None:\n", + " return None\n", + " else:\n", + " return jnp.not_equal(inputs, 0)" + ] + }, + { + "cell_type": "markdown", + "id": "861be55c", + "metadata": {}, + "source": [ + "`TransformerDecoder` implements the decoder block with two attention layers:\n", + "- `attention_1` is masked self-attention over the target sequence. A causal mask prevents each position from attending to future tokens, which is essential so the model can only use tokens it has already generated when predicting the next one.\n", + "- `attention_2` is cross-attention where queries come from the decoder and keys/values come from the encoder output, letting the decoder attend to the full source sentence at every step.\n", + "\n", + "Each attention layer is followed by a residual connection, layer norm, and a shared feed-forward projection." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "121bf138-34b3-4be9-a0fc-3bbac81f469a", + "metadata": {}, + "outputs": [], + "source": [ + "class TransformerDecoder(nnx.Module):\n", + " def __init__(self, embed_dim: int, latent_dim: int, num_heads: int, rngs: nnx.Rngs, **kwargs):\n", + " self.embed_dim = embed_dim\n", + " self.latent_dim = latent_dim\n", + " self.num_heads = num_heads\n", + " self.attention_1 = nnx.MultiHeadAttention(num_heads=num_heads,\n", + " in_features=embed_dim,\n", + " decode=False,\n", + " rngs=rngs)\n", + " self.attention_2 = nnx.MultiHeadAttention(num_heads=num_heads,\n", + " in_features=embed_dim,\n", + " decode=False,\n", + " rngs=rngs)\n", + "\n", + " self.dense_proj = nnx.Sequential(\n", + " nnx.Linear(embed_dim, latent_dim, rngs=rngs),\n", + " nnx.relu,\n", + " nnx.Linear(latent_dim, embed_dim, rngs=rngs),\n", + " )\n", + " self.layernorm_1 = nnx.LayerNorm(embed_dim, rngs=rngs)\n", + " self.layernorm_2 = nnx.LayerNorm(embed_dim, rngs=rngs)\n", + " self.layernorm_3 = nnx.LayerNorm(embed_dim, rngs=rngs)\n", + "\n", + " def __call__(self, inputs, encoder_outputs):\n", + " causal_mask = nnx.make_causal_mask(inputs[:,:,0])\n", + " attention_output_1 = self.attention_1(\n", + " inputs_q=inputs, inputs_v=inputs, inputs_k=inputs, mask=causal_mask\n", + " )\n", + " out_1 = self.layernorm_1(inputs + attention_output_1)\n", + "\n", + " attention_output_2 = self.attention_2(\n", + " inputs_q=out_1,\n", + " inputs_v=encoder_outputs,\n", + " inputs_k=encoder_outputs\n", + " )\n", + " out_2 = self.layernorm_2(out_1 + attention_output_2)\n", + "\n", + " proj_output = self.dense_proj(out_2)\n", + " return self.layernorm_3(out_2 + proj_output)" + ] + }, + { + "cell_type": "markdown", + "id": "d033ae31-cc43-4e61-8d7f-cdc6d55b8bf9", + "metadata": {}, + "source": [ + "`TransformerModel` wires all the components together into the full encoder-decoder architecture. The same `positional_embedding` layer is reused for both the source and target sequences. The forward pass:\n", + "1. Embeds and positionally encodes the English (`encoder_inputs`) tokens and passes them through the encoder.\n", + "2. Embeds and positionally encodes the Spanish (`decoder_inputs`) tokens, passes them through the decoder along with the encoder output, and applies dropout.\n", + "3. Projects the decoder output to a distribution over the vocabulary with a final linear layer." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "c5dcfaf6-f5cd-40f4-bbf0-2754c0193327", + "metadata": {}, + "outputs": [], + "source": [ + "class TransformerModel(nnx.Module):\n", + " def __init__(self, sequence_length: int, vocab_size: int, embed_dim: int, latent_dim: int, num_heads: int, dropout_rate: float, rngs: nnx.Rngs):\n", + " self.sequence_length = sequence_length\n", + " self.vocab_size = vocab_size\n", + " self.embed_dim = embed_dim\n", + " self.latent_dim = latent_dim\n", + " self.num_heads = num_heads\n", + " self.dropout_rate = dropout_rate\n", + "\n", + " self.encoder = TransformerEncoder(embed_dim, latent_dim, num_heads, rngs=rngs)\n", + " self.positional_embedding = PositionalEmbedding(sequence_length, vocab_size, embed_dim, rngs=rngs)\n", + " self.decoder = TransformerDecoder(embed_dim, latent_dim, num_heads, rngs=rngs)\n", + " self.dropout = nnx.Dropout(rate=dropout_rate, rngs=rngs)\n", + " self.dense = nnx.Linear(embed_dim, vocab_size, rngs=rngs)\n", + "\n", + " def __call__(self, encoder_inputs: jnp.array, decoder_inputs: jnp.array):\n", + " x = self.positional_embedding(encoder_inputs)\n", + " encoder_outputs = self.encoder(x)\n", + "\n", + " x = self.positional_embedding(decoder_inputs)\n", + " decoder_outputs = self.decoder(x, encoder_outputs)\n", + " decoder_outputs = self.dropout(decoder_outputs, deterministic=False)\n", + "\n", + " logits = self.dense(decoder_outputs)\n", + " return logits" + ] + }, + { + "cell_type": "markdown", + "id": "1744cd95-afcc-4a82-9a00-18fef4f6f7df", + "metadata": {}, + "source": [ + "## Build out Data Loader and Training Definitions\n", + "It can be more computationally efficient to use pygrain for the data load stage, but this way it's abundandtly clear what's happening: data pairs go in and sets of jnp arrays come out, in step with our original dictionaries. 'Encoder_inputs', 'decoder_inputs' and 'target_output'." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "1fb8cb44-9012-4802-9286-1efc19dd2ba1", + "metadata": {}, + "outputs": [], + "source": [ + "batch_size = 512 #set here for the loader and model train later on\n", + "\n", + "class CustomPreprocessing(grain.MapTransform):\n", + " def __init__(self):\n", + " pass\n", + "\n", + " def map(self, data):\n", + " return {\n", + " \"encoder_inputs\": np.array(data[\"encoder_inputs\"]),\n", + " \"decoder_inputs\": np.array(data[\"decoder_inputs\"]),\n", + " \"target_output\": np.array(data[\"target_output\"]),\n", + " }\n", + "\n", + "train_sampler = grain.IndexSampler(\n", + " len(train_data),\n", + " shuffle=True,\n", + " seed=12, # Seed for reproducibility\n", + " shard_options=grain.NoSharding(), # No sharding since it's a single-device setup\n", + " num_epochs=1, # Iterate over the dataset for one epoch\n", + ")\n", + "\n", + "val_sampler = grain.IndexSampler(\n", + " len(val_data),\n", + " shuffle=False,\n", + " seed=12,\n", + " shard_options=grain.NoSharding(),\n", + " num_epochs=1,\n", + ")\n", + "\n", + "train_loader = grain.DataLoader(\n", + " data_source=train_data,\n", + " sampler=train_sampler, # Sampler to determine how to access the data\n", + " worker_count=4, # Number of child processes launched to parallelize the transformations\n", + " worker_buffer_size=2, # Count of output batches to produce in advance per worker\n", + " operations=[\n", + " CustomPreprocessing(),\n", + " grain.Batch(batch_size=batch_size, drop_remainder=True),\n", + " ]\n", + ")\n", + "\n", + "val_loader = grain.DataLoader(\n", + " data_source=val_data,\n", + " sampler=val_sampler,\n", + " worker_count=4,\n", + " worker_buffer_size=2,\n", + " operations=[\n", + " CustomPreprocessing(),\n", + " grain.Batch(batch_size=batch_size),\n", + " ]\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "40d9707d-a73c-47f5-8c12-1f336e526e61", + "metadata": {}, + "source": [ + "Optax doesn't have the identical loss function that the source tutorial uses, but this softmax cross entropy works well here - you can one_hot_encode if you don't use the `_with_integer_labels` version of the loss." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "d2f8e06f-1126-41cc-b8d8-de6bd7a5255a", + "metadata": {}, + "outputs": [], + "source": [ + "def compute_loss(logits, labels):\n", + " loss = optax.softmax_cross_entropy_with_integer_labels(logits=logits, labels=labels)\n", + " return jnp.mean(loss)" + ] + }, + { + "cell_type": "markdown", + "id": "0a1b625a-d9e7-4028-bc98-521ce1632450", + "metadata": {}, + "source": [ + "While in the original tutorial most of the model and training details happen inside keras, we make them explicit here in our step functions, which are later used in `train_one_epoch` and `evaluate_model`.\n", + "\n", + "`train_step` runs a single forward pass, computes the loss, calculates gradients with `nnx.value_and_grad`, and applies them via the optimizer. It is JIT-compiled for performance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ac62d2ce", + "metadata": {}, + "outputs": [], + "source": [ + "@nnx.jit\n", + "def train_step(model, optimizer, batch):\n", + " def loss_fn(model, train_encoder_input, train_decoder_input, train_target_input):\n", + " logits = model(train_encoder_input, train_decoder_input)\n", + " loss = compute_loss(logits, train_target_input)\n", + " return loss\n", + "\n", + " grad_fn = nnx.value_and_grad(loss_fn)\n", + " loss, grads = grad_fn(model, jnp.array(batch[\"encoder_inputs\"]), jnp.array(batch[\"decoder_inputs\"]), jnp.array(batch[\"target_output\"]))\n", + " optimizer.update(grads)\n", + " return loss" + ] + }, + { + "cell_type": "markdown", + "id": "991df2a9", + "metadata": {}, + "source": [ + "`eval_step` runs a forward pass without updating weights and accumulates loss and accuracy into `eval_metrics`:" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "279d991f-f129-48b3-9b7e-d143019c18a8", + "metadata": {}, + "outputs": [], + "source": [ + "@nnx.jit\n", + "def eval_step(model, batch, eval_metrics):\n", + " logits = model(jnp.array(batch[\"encoder_inputs\"]), jnp.array(batch[\"decoder_inputs\"]))\n", + " loss = compute_loss(logits, jnp.array(batch[\"target_output\"]))\n", + " labels = jnp.array(batch[\"target_output\"])\n", + "\n", + " eval_metrics.update(\n", + " loss=loss,\n", + " logits=logits,\n", + " labels=labels,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "04e53ee9-6da1-431c-8b3f-f619d3fee68f", + "metadata": {}, + "source": [ + "`nnx.MultiMetric` accumulates loss and accuracy across batches. Separate history dictionaries record per-epoch values for later plotting:" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "32a17edc-33d0-41bc-a516-8b8ce45c3ad7", + "metadata": {}, + "outputs": [], + "source": [ + "eval_metrics = nnx.MultiMetric(\n", + " loss=nnx.metrics.Average('loss'),\n", + " accuracy=nnx.metrics.Accuracy(),\n", + ")\n", + "\n", + "train_metrics_history = {\n", + " \"train_loss\": [],\n", + "}\n", + "\n", + "eval_metrics_history = {\n", + " \"test_loss\": [],\n", + " \"test_accuracy\": [],\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "a46b51f8", + "metadata": {}, + "source": [ + "Key hyperparameters for the model and training run:\n", + "- `embed_dim` — dimensionality of token and position embeddings.\n", + "- `latent_dim` — width of the feed-forward projection inside each encoder/decoder block.\n", + "- `num_heads` — number of attention heads.\n", + "- `dropout_rate` — fraction of activations dropped during training.\n", + "- `learning_rate` / `num_epochs` — AdamW step size and training duration." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "1189a6a6-2cc6-4c87-9f87-b4b800a1513d", + "metadata": {}, + "outputs": [], + "source": [ + "## Hyperparameters\n", + "rng = nnx.Rngs(0)\n", + "embed_dim = 256\n", + "latent_dim = 2048\n", + "num_heads = 8\n", + "dropout_rate = 0.5\n", + "vocab_size = tokenizer.n_vocab\n", + "sequence_length = 20\n", + "learning_rate = 1.5e-3\n", + "num_epochs = 10" + ] + }, + { + "cell_type": "markdown", + "id": "b738a87e", + "metadata": {}, + "source": [ + "Although we'll want full dropout randomization for training, we'll want to evaluate our model without dropout by setting the `deterministic=True` flag. We'll make two views of our model (`train_model` and `eval_model`): one with each flag setting. Both views share the same underlying parameters, so a weight update through `train_model` is immediately visible when running `eval_model`." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "fbeb6101-be11-4a33-9650-a3efd3656855", + "metadata": {}, + "outputs": [], + "source": [ + "model = TransformerModel(sequence_length, vocab_size, embed_dim, latent_dim, num_heads, dropout_rate, rngs=rng)\n", + "train_model = nnx.view(model, deterministic=False)\n", + "eval_model = nnx.view(model, deterministic=True)\n", + "\n", + "optimizer = nnx.ModelAndOptimizer(model, optax.adamw(learning_rate))" + ] + }, + { + "cell_type": "markdown", + "id": "825bfe8b", + "metadata": {}, + "source": [ + "`train_one_epoch` iterates over all training batches, calls `train_step` on each, and logs the per-step loss. `evaluate_model` resets the accumulated metrics, runs `eval_step` over the entire validation set, then prints and records the epoch-level loss and accuracy:" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "49a1d33a-c2e4-4d48-821b-519f5c0192c7", + "metadata": {}, + "outputs": [], + "source": [ + "bar_format = \"{desc}[{n_fmt}/{total_fmt}]{postfix} [{elapsed}<{remaining}]\"\n", + "train_total_steps = len(train_data) // batch_size\n", + "\n", + "def train_one_epoch(epoch):\n", + " with tqdm.tqdm(\n", + " desc=f\"[train] epoch: {epoch}/{num_epochs}, \",\n", + " total=train_total_steps,\n", + " bar_format=bar_format,\n", + " leave=True,\n", + " ) as pbar:\n", + " for batch in train_loader:\n", + " loss = train_step(train_model, optimizer, batch)\n", + " train_metrics_history[\"train_loss\"].append(loss.item())\n", + " pbar.set_postfix({\"loss\": loss.item()})\n", + " pbar.update(1)\n", + "\n", + "\n", + "def evaluate_model(epoch):\n", + " # Compute the metrics on the train and val sets after each training epoch.\n", + "\n", + "\n", + " eval_metrics.reset() # Reset the eval metrics\n", + " for val_batch in val_loader:\n", + " eval_step(eval_model, val_batch, eval_metrics)\n", + "\n", + " for metric, value in eval_metrics.compute().items():\n", + " eval_metrics_history[f'test_{metric}'].append(value)\n", + "\n", + " print(f\"[test] epoch: {epoch + 1}/{num_epochs}\")\n", + " print(f\"- total loss: {eval_metrics_history['test_loss'][-1]:0.4f}\")\n", + " print(f\"- Accuracy: {eval_metrics_history['test_accuracy'][-1]:0.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "fa7d5601-60c1-4131-a40c-c670f055ce68", + "metadata": {}, + "source": [ + "## Start the Training!\n", + "With our data loaders set and the model, optimizer, and epoch train/eval functions set up - time to finally press go - on a 3090, this is roughly 19GB VRAM and each epoch is roughly 18 seconds with batch_size set to 512." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "c764510c-4d98-46ad-b877-8cfc2fa5a9ea", + "metadata": { + "outputId": "a4f97db4-da4e-481b-f5c2-7137216e6380" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 0/10, [160/162], loss=1.96 [00:49<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 1/10\n", + "- total loss: 1.9830\n", + "- Accuracy: 0.6765\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 1/10, [160/162], loss=1.14 [00:09<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 2/10\n", + "- total loss: 1.2113\n", + "- Accuracy: 0.7875\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 2/10, [160/162], loss=0.83 [00:09<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 3/10\n", + "- total loss: 1.0127\n", + "- Accuracy: 0.8173\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 3/10, [160/162], loss=0.676 [00:10<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 4/10\n", + "- total loss: 0.9345\n", + "- Accuracy: 0.8298\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 4/10, [160/162], loss=0.575 [00:09<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 5/10\n", + "- total loss: 0.9091\n", + "- Accuracy: 0.8346\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 5/10, [160/162], loss=0.512 [00:09<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 6/10\n", + "- total loss: 0.8982\n", + "- Accuracy: 0.8394\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 6/10, [160/162], loss=0.445 [00:10<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 7/10\n", + "- total loss: 0.9035\n", + "- Accuracy: 0.8393\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 7/10, [160/162], loss=0.425 [00:09<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 8/10\n", + "- total loss: 0.9187\n", + "- Accuracy: 0.8418\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 8/10, [160/162], loss=0.393 [00:09<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 9/10\n", + "- total loss: 0.9357\n", + "- Accuracy: 0.8421\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[train] epoch: 9/10, [160/162], loss=0.359 [00:09<00:00]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[test] epoch: 10/10\n", + "- total loss: 0.9405\n", + "- Accuracy: 0.8408\n" + ] + } + ], + "source": [ + "for epoch in range(num_epochs):\n", + " train_one_epoch(epoch)\n", + " evaluate_model(epoch)" + ] + }, + { + "cell_type": "markdown", + "id": "f922eac4-8338-4a0d-bc6d-1f5880079bde", + "metadata": {}, + "source": [ + "We can then plot the loss over training time. That log-plot comes in handy here, or it's hard to appreciate the progress after 1000 steps or so." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "a79ecfa5-d74a-4956-9ee2-cbed86d5a82f", + "metadata": { + "outputId": "c1cdf060-335f-4321-8982-7ed07d15db06" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAAGdCAYAAADJ6dNTAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAVw9JREFUeJzt3XlcVFXjBvBnZth3WWRHUHFBETcwt9wotzKzxcxMzTLfsDTN1J9pq0tZZhrZrr6VafWamrvhriiIiBu4oqCyqMi+M/f3B3CZgWF1mDsDz/fz4fPO3Hvm3nPMl3k89ywyQRAEEBEREekJudQVICIiIlLFcEJERER6heGEiIiI9ArDCREREekVhhMiIiLSKwwnREREpFcYToiIiEivMJwQERGRXjGSugL1pVQqcefOHVhbW0Mmk0ldHSIiIqoDQRCQlZUFNzc3yOU1940YXDi5c+cOPD09pa4GERERNUBiYiI8PDxqLGNw4cTa2hpAaeNsbGwkrg0RERHVRWZmJjw9PcXv8ZoYXDgpf5RjY2PDcEJERGRg6jIkgwNiiYiISK8wnBAREZFeYTghIiIivWJwY06ISP+VlJSgqKhI6moQkQ4pFAoYGRlpZZkPhhMi0qrs7GzcunULgiBIXRUi0jELCwu4urrCxMTkoa7DcEJEWlNSUoJbt27BwsICTk5OXCiRqJkQBAGFhYW4e/cu4uPj4evrW+tCazVhOCEirSkqKoIgCHBycoK5ubnU1SEiHTI3N4exsTFu3ryJwsJCmJmZNfhaHBBLRFrHHhOi5ulhekvUrqOVqxARERFpCcMJEVETcvDgQchkMqSnp+v83gMHDsTMmTMf+jrr1q2DnZ3dQ1+noWQyGbZs2SLZ/evigw8+QNeuXev1GW9vb6xcubJR6qNtDCdE1OxNmjQJo0ePlroaVGbs2LG4fPlyo9+nIV/wDaXtwPXOO+8gLCysXp+JjIzE1KlTtVaHxsQBsUREpDeKiopgbm7ebAdUFxYW1mkarpWVFaysrOp1bScnp4ZWS+fYc1Lm8OW7+GDbBWyLuSN1VYhIzxw6dAhBQUEwNTWFq6sr5s2bh+LiYvH8X3/9BX9/f5ibm8PBwQHBwcHIyckBUPqYJSgoCJaWlrCzs0Pfvn1x8+ZNjffp06cP5s6dq3bs7t27MDY2xuHDhwEAv/zyC3r27Alra2u4uLjgxRdfRGpqarV119Q7sHLlSnh7e6sd+/HHH9GxY0eYmZmhQ4cO+Oabb2r8M8nJycHLL78MKysruLq64osvvqhSRtPjETs7O6xbtw4AcOPGDchkMmzatAkDBgyAmZkZfvvttyq9DOVt+OWXX+Dt7Q1bW1u88MILyMrKEstkZWVh/PjxsLS0hKurK7788ssaHzOtW7cOH374IWJiYiCTySCTycR6AcC9e/fw9NNPw8LCAr6+vti2bZva58+fP4/hw4fDysoKzs7OmDBhAu7du6fxXgcPHsTkyZORkZEh3uuDDz4AUPqo5eOPP8bLL78MGxsbsWdj7ty5aNeuHSwsLNC6dWssXLhQbWHDyv9dy3v/Pv/8c7i6usLBwQEhISFqn6n8WEcmk+HHH3+ssZ3btm2Dr68vzMzMMGjQIKxfv14njw0ZTsqcvZWOdcdvIPya5r9cRFR/giAgt7BYkh9tLQJ3+/ZtjBgxAoGBgYiJicGaNWvw008/4ZNPPgEAJCUlYdy4cXjllVcQGxuLgwcPYsyYMRAEAcXFxRg9ejQGDBiAs2fPIjw8HFOnTq12NtP48eOxceNGtbpv2rQJbm5u6N+/P4DSnoWPP/4YMTEx2LJlC27cuIFJkyY9VBt/++03LFq0CIsXL0ZsbCyWLFmChQsXYv369dV+Zs6cOTh06BC2bt2KvXv34uDBgzh9+nSD7j9v3jzMmDEDsbGxGDp0qMYy165dw5YtW7B9+3Zs374dhw4dwrJly8Tzs2bNwrFjx7Bt2zbs27cPR44cqbE+Y8eOxezZs9GpUyckJSUhKSkJY8eOFc9/+OGHeP7553H27FmMGDEC48ePR1paGgAgPT0dgwcPRrdu3XDq1Cns3r0bKSkpeP755zXeq0+fPli5ciVsbGzEe73zzjvi+c8//xwBAQGIjo7GwoULAQDW1tZYt24dLl68iK+++go//PADvvzyyxr/HA8cOIBr167hwIEDWL9+PdatW6cWuDSpqZ3x8fF49tlnMXr0aMTExOD111/HggULaryetvCxTpnyXxZKpcQVIWpC8opK4LdojyT3vvjRUFiYPPyvuG+++Qaenp74+uuvIZPJ0KFDB9y5cwdz587FokWLkJSUhOLiYowZMwatWrUCAPj7+wMA0tLSkJGRgSeeeAJt2rQBAHTs2LHaez3//POYOXMmjh49KoaRDRs2YNy4ceLvqFdeeUUs37p1a6xatQqBgYHIzs6udzd/uffffx9ffPEFxowZAwDw8fHBxYsX8d1332HixIlVymdnZ+Onn37Cr7/+iiFDhgAA1q9fDw8Pjwbdf+bMmeK9q6NUKrFu3TpYW1sDACZMmICwsDAsXrwYWVlZWL9+PTZs2CDWZ+3atXBzc6v2eubm5rCysoKRkRFcXFyqnJ80aRLGjRsHAFiyZAlWrVqFiIgIDBs2DF9//TW6deuGJUuWiOV//vlneHp64vLly2jXrp3atUxMTGBrawuZTKbxXoMHD8bs2bPVjr333nvia29vb7zzzjvYuHEj3n333Wrb1KJFC3z99ddQKBTo0KEDRo4cibCwMLz22mvVfqamdn733Xdo3749li9fDgBo3749zp8/j8WLF1d7PW1hz0mZ8n/IKLnkNhGpiI2NRe/evdV6O/r27Ssu0x8QEIAhQ4bA398fzz33HH744Qc8ePAAAGBvb49JkyZh6NChePLJJ/HVV18hKSmp2ns5OTnh8ccfx2+//Qag9F+u4eHhGD9+vFgmKioKTz75JLy8vGBtbY0BAwYAABISEhrUvpycHFy7dg1TpkwRxzFYWVnhk08+wbVr1zR+5tq1aygsLESvXr3EY/b29mjfvn2D6tCzZ89ay3h7e4vBBABcXV3Fx1nXr19HUVERgoKCxPO2trYNrg8AdOnSRXxtaWkJGxsb8X4xMTE4cOCA2p9Xhw4dAKDaP7OaaGr/pk2b0LdvX7i4uMDKygrvvfderf+NO3XqBIVCIb5X/TOqTk3tvHTpEgIDA9XKq/4ZNyb2nJSRl/3iYTQh0h5zYwUufqS5m14X99YFhUKBffv24fjx49i7dy9Wr16NBQsW4OTJk/Dx8cHatWvx1ltvYffu3di0aRPee+897Nu3D4888ojG640fPx5vvfUWVq9ejQ0bNsDf31/sicnJycHQoUMxdOhQ/Pbbb3ByckJCQgKGDh2KwsJCjdeTy+VVHnGpjkPIzs4GAPzwww9qYaO8bQ9DJpPVeO9ylpaWtV7L2Ni4yrWVjdjVXdP9srOz8eSTT+LTTz+t8jlXV9d636ty+8sD6YcffoihQ4fC1tYWGzdu1Diup6511uZndIE9J2Xk7Dkh0jqZTAYLEyNJfrS1Sm3Hjh0RHh6u9iV77NgxWFtbi48xZDIZ+vbtiw8//BDR0dEwMTHB33//LZbv1q0b5s+fj+PHj6Nz587YsGFDtfd76qmnkJ+fj927d2PDhg1qvSZxcXG4f/8+li1bhv79+6NDhw61/svYyckJycnJavU/c+aM+NrZ2Rlubm64fv062rZtq/bj4+Oj8Zpt2rSBsbExTp48KR578OBBlem/Tk5Oaj1FV65cQW5ubo31bYjWrVvD2NgYkZGR4rGMjIxapyObmJigpKSk3vfr3r07Lly4AG9v7yp/ZtUFrfrc6/jx42jVqhUWLFiAnj17wtfXt9pB1I2pffv2OHXqlNox1T/jxsRwUkaGsp4TZhOiZikjIwNnzpxR+0lMTMQbb7yBxMREvPnmm4iLi8PWrVvx/vvvY9asWZDL5Th58iSWLFmCU6dOISEhAZs3b8bdu3fRsWNHxMfHY/78+QgPD8fNmzexd+9eXLlypcZxJ5aWlhg9ejQWLlyI2NhYcTwAAHh5ecHExASrV6/G9evXsW3bNnz88cc1tmvgwIG4e/cuPvvsM1y7dg2hoaHYtWuXWpkPP/wQS5cuxapVq3D58mWcO3cOa9euxYoVKzRe08rKClOmTMGcOXOwf/9+nD9/HpMmTaqydPngwYPx9ddfIzo6GqdOncK0adOq/EtdG6ytrTFx4kTMmTMHBw4cwIULFzBlyhTI5fIaQ6q3tzfi4+Nx5swZ3Lt3DwUFBXW6X0hICNLS0jBu3DhERkbi2rVr2LNnDyZPnlxtAPH29kZ2djbCwsJw7969GkOar68vEhISsHHjRly7dg2rVq1SC7u68vrrryMuLg5z587F5cuX8ccff4gDbBt7iwqGkzIcc0LUvB08eBDdunVT+/nwww/h7u6OnTt3IiIiAgEBAZg2bRqmTJkiDli0sbHB4cOHMWLECLRr1w7vvfcevvjiCwwfPhwWFhaIi4vDM888g3bt2mHq1KkICQnB66+/XmNdxo8fj5iYGPTv3x9eXl7icScnJ6xbtw5//vkn/Pz8sGzZMnz++ec1Xqtjx4745ptvEBoaioCAAERERKjNFAGAV199FT/++CPWrl0Lf39/DBgwAOvWrau25wQAli9fjv79++PJJ59EcHAw+vXrhx49eqiV+eKLL+Dp6Yn+/fvjxRdfxDvvvAMLC4sa69tQK1asQO/evfHEE08gODgYffv2FadGV+eZZ57BsGHDMGjQIDg5OeH333+v073c3Nxw7NgxlJSU4PHHH4e/vz9mzpwJOzu7aveW6dOnD6ZNm4axY8fCyckJn332WbXXHzVqFN5++21Mnz4dXbt2xfHjx8VZPLrk4+ODv/76C5s3b0aXLl2wZs0acbaOqalpo95bJmhrvp2OZGZmwtbWFhkZGbCxsdHadX8+Go+Ptl/EqAA3rBrXTWvXJWpO8vPzER8fDx8fn4fakZToYeXk5MDd3R1ffPEFpkyZInV1mozFixfj22+/RWJiosbzNf0OqM/3NwfElmHPCRGR4YqOjkZcXByCgoKQkZGBjz76CEDpGB5quG+++QaBgYFwcHDAsWPHsHz5ckyfPr3R78twUkacrcNsQkRkkD7//HNcunQJJiYm6NGjB44cOQJHR0epq2XQrly5gk8++QRpaWnw8vLC7NmzMX/+/Ea/L8NJmfLZOgInExMRGZxu3bohKipK6mo0OV9++WWtK9M2Bg6ILcMVYomIiPQDw0kZjjkhIiLSDwwnZbhCLJH2GNgkQCLSEm39f1+ScPL000+jRYsWePbZZ6W4vUbimBP+UiVqsPLlzqtbSp2ImrbyxeUedrE9SQbEzpgxA6+88kqN23HrWvkKsUpmE6IGMzIygoWFBe7evQtjY+NqF6QioqZFEATk5uYiNTUVdnZ2D70vkyThZODAgTh48KAUt66WjD0nRA9NJpPB1dUV8fHxkuwFQkTSsrOzg4uLy0Nfp97h5PDhw1i+fDmioqKQlJSEv//+G6NHj1YrExoaiuXLlyM5ORkBAQFYvXq1zrZZbqjyMSfsOSF6OCYmJvD19eWjHaJmxtjY+KF7TMrVO5zk5OQgICAAr7zyCsaMGVPl/KZNmzBr1ix8++236NWrF1auXImhQ4fi0qVLaNmypVYq3Rg4W4dIe+RyOZevJ6IGq3c4GT58OIYPH17t+RUrVuC1117D5MmTAQDffvstduzYgZ9//hnz5s2rdwULCgrUdorMzMys9zXqQt7IOywSERFR3Wh1tFphYSGioqIQHBxccQO5HMHBwQgPD2/QNZcuXQpbW1vxx9PTU1vVVcOeEyIiIv2g1XBy7949lJSUwNnZWe24s7MzkpOTxffBwcF47rnnsHPnTnh4eNQYXObPn4+MjAzxp7qdEB8WV4glIiLSD5LM1vn333/rXNbU1BSmpqaNWJtS3FuHiIhIP2i158TR0REKhQIpKSlqx1NSUrQytagxcbYOERGRftBqOCnfpjosLEw8plQqERYWht69e2vzVlpXPhyW65wQERFJq96PdbKzs3H16lXxfXx8PM6cOQN7e3t4eXlh1qxZmDhxInr27ImgoCCsXLkSOTk54uwdfVU+5oTZhIiISFr1DienTp3CoEGDxPezZs0CAEycOBHr1q3D2LFjcffuXSxatAjJycno2rUrdu/eXWWQrL6Rc7YOERGRXqh3OBk4cGCtjz6mT5+O6dOnN7hSUpBxzAkREZFe4K5cZSpm6xAREZGUDCachIaGws/PD4GBgY1yfbk45oTxhIiISEoGE05CQkJw8eJFREZGNs4NOOaEiIhILxhMOGlscs7WISIi0gsMJ2UqZutIWw8iIqLmjuGkjAwcc0JERKQPGE7KiLN1mE2IiIgkxXBSpmKdE6YTIiIiKTGclJFxtg4REZFeYDgpI87WkbgeREREzR3DSRmOOSEiItIPDCdl+FiHiIhIPxhMOGns5etlXISNiIhILxhMOGns5evlnK1DRESkFwwmnDS2sqc67DkhIiKSGMNJGfacEBER6QeGkzImRqV/FIXFSolrQkRE1LwxnJQxMy79o8gvKpG4JkRERM0bw0kZM2MFACCfPSdERESSYjgpY1r2WKdEKaC4hAGFiIhIKgwnZcp7TgD2nhAREUmJ4aSMiaLij6KA406IiIgkw3BSRi6XiTN22HNCREQkHYYTFeXjTthzQkREJB2GExXijJ0i9pwQERFJxWDCSWNv/AdU9JzkF7PnhIiISCoGE04ae+M/oKLnpIA9J0RERJIxmHCiC+Iqsew5ISIikgzDiQpTI/acEBERSY3hREV5z0kBe06IiIgkw3Cigj0nRERE0mM4UcExJ0RERNJjOFFR3nOSz0XYiIiIJMNwokIcc8LHOkRERJJhOFEh9pzwsQ4REZFkGE5UmLLnhIiISHIMJyrM2HNCREQkOYYTFew5ISIikh7DiYqKnhOGEyIiIqkYTDjRya7E5euccCoxERGRZAwmnOhkV+LyFWLZc0JERCQZgwknumBmzEXYiIiIpMZwosLUqHzjP/acEBERSYXhREV5z0kBe06IiIgkw3CiQpxKzJ4TIiIiyTCcqDDjxn9ERESSYzhRwanERERE0mM4UcGpxERERNJjOFGh2nMiCILEtSEiImqeGE5UlPecKAWgqIThhIiISAoMJyrMTRTi67xCjjshIiKSAsOJChMjOYzkMgBAblGxxLUhIiJqnhhOKinvPcllzwkREZEkGE4qsSgLJ3ysQ0REJA2Gk0osTIwAsOeEiIhIKgwnlViIj3U45oSIiEgKBhNOQkND4efnh8DAwEa9jwXHnBAREUnKYMJJSEgILl68iMjIyEa9jzkf6xAREUnKYMKJrlgYlw+I5WMdIiIiKTCcVMLHOkRERNJiOKmE65wQERFJi+GkEnGdkyKGEyIiIikwnFRSvs5JTgHHnBAREUmB4aQSa7PScJLNcEJERCQJhpNKysNJVj7DCRERkRQYTiqxMjUGAGQznBAREUmC4aSS8p6TzPwiiWtCRETUPDGcVGLFMSdERESSYjipxIZjToiIiCTFcFKJtVnZmJOCYgiCIHFtiIiImh+Gk0qsTEt7TkqUAhdiIyIikgDDSSUWJgrIZaWvOWOHiIhI9xhOKpHJZGLvSSbDCRERkc4xnGigOu6EiIiIdIvhRIOKVWK51gkREZGuMZxoIC7ElseeEyIiIl1jONEgNasAADBv81mJa0JERNT8GEw4CQ0NhZ+fHwIDAxv9XubGCgBciI2IiEgKBhNOQkJCcPHiRURGRjb6vb54PgBAxeMdIiIi0h2DCSe61MrBEkBpz0kOZ+wQERHpFMOJBpYmCvHRzr3sAolrQ0RE1LwwnGggk8ngZG0KALibxXBCRESkSwwn1WA4ISIikgbDSTValocTPtYhIiLSKYaTapT3nKRmMpwQERHpEsNJNTxamAMA4u/lSFwTIiKi5oXhpBrtnK0BAFdTsyWuCRERUfPCcFINR6vSxzoZedz8j4iISJcYTqpRvjpscmY+4pIzJa4NERFR88FwUg1rM2Px9aItF3DuVgYEQZCwRkRERM0Dw0k1VPfVibiRhie/PorX/hslYY2IiIiaB4aTahgrqv7R/BubIkFNiIiImheGkxo8092jyrHiEqUENSEiImo+GE5qkJZTdQG2zHzuUkxERNSYGE5qoJDLqhz741SiBDUhIiJqPhhOajBnaIcqx5btipOgJkRERM0Hw0kN2rtY4zE/5yrHl+6Mxa8nbnL8CRERUSMwqr1I87ZmfHdkFxTDSCFH5/f3AAC+O3wdQOng2El9faSsHhERUZPDnpNaGCnksLMwgZWpEUIGtVE798E/F5GZz+XtiYiItInhpB783e2qHItLytJ9RYiIiJowhpN60LR8/e30XAlqQkRE1HQxnNRDF0+7KsduP8jTfUWIiIiaMIaTenC3M8e/sx7FZ892EY/dTs9HYbESJUpuCkhERKQNnK1TT21bWqNtS2vkF5Vg0dYL+D0iAb9HJKCrpx22hPSVunpEREQGz2B6TkJDQ+Hn54fAwECpqwIAaONkpfb+TGI68otKJKoNERFR02Ew4SQkJAQXL15EZGSk1FUBUPqIp7L0XE4rJiIielgGE070jae9RZVjsUmZHHtCRET0kBhOGkghl+HyJ8PVjk1eF4nAxf/iXnbV3YyJiIiobhhOHoKJkRw/TeypdiwtpxB/Rd2SqEZERESGj+HkIQ3p6IxV47qpHfsfwwkREVGDMZxowaD2Tmrvr6RmIzYpU6LaEBERGTaGEy2wNjOGQi5TOzb8qyO4nc7VY4mIiOqL4URLLnw4tMqxV9bqx7RnIiIiQ8JwoiVmxgqEzR6gduxSCncsJiIiqi+GEy1q42SFb8Z3l7oaREREBo3hRMtG+Lvi6NxB4vt5/zuLq6nsQSEiIqorhpNG4GZbsbT9xshEBK84jKISpYQ1IiIiMhwMJ41ALpfh50nqi7N9vveSRLUhIiIyLAwnjaS/r/raJ98dus7eEyIiojpgOGkkxoqqf7S+C3Zh+Z44CWpDRERkOBhOGtH3E3pUORZ64BpWh11B+LX7EtSIiIhI/8kEQRCkrkR9ZGZmwtbWFhkZGbCxsZG6OnXiPW+HxuMvBHoiZFBbeNpb6LhGREREulWf72/2nOhAkI+9xuMbIxPx2n9P6bg2RERE+o3hRAe+e6kHvnqhq8ZzcclcA4WIiEgVw4kOtLA0wVNd3as9X1yiRPi1+4i/l6PDWhEREeknI6krQEDbBbvE1zeWjQQACIKAzLxi2FoYS1UtIiIiSbDnRIfmDG1fa5l/Yu4AAJbsjEXAR3tx4FJqY1eLiIhIrzCc6NBIf9day7z5ezRuPcjFD0fiAQBLdsQ2drWIiIj0Ch/r6JC3oyU+eNIPLSxNcC01G6v2X9VYrt+nB8TXcplMV9UjIiLSCwwnOjapr4/4ekq/1lhz6Bq+PXSt2vLMJkRE1NzwsY6EbC2MMW94B4TPH1xtGRnTCRERNTMMJ3rAxcas2nOxSZk6rAkREZH0GE70QG29I+UzeIiIiJoDhhM9MzPYF94O6nvtvPl7NFaHXZGoRkRERLrFcKJngjs648A7A6sc/2LfZRSVKHVfISIiIh1jONETYbMH4JcpQejsbguZTIZA7xZVyvgu2IVfTtxEem4htp65jfyiEglqSkRE1LhkgiAIUleiPuqz5bIhS87Ix5y/YnDkyr0q5xRyGUqUAib18cYHozpJUDsiIqL6qc/3N3tO9JSLrRl+mdJL47kSZWmeXHf8hg5rREREpBsMJ0RERKRXGE703J6Zj8LUqPr/TBsjEnRYGyIiosbHcKLn2rtY49i86leQVV36vrhEiYJiDpIlIiLDxnBiABytTPHntN7i+z+n9caHZQNhb6fnQVk2BuW578LR/9MDnMVDREQGzWDCSWhoKPz8/BAYGCh1VSSRW1gROLp7tcD4Xl5QyGUoKhFw9W42SpQCohPSkZpVgJjEdOkqSkRE9JAMJpyEhITg4sWLiIyMlLoqkujsVjrtytrUCAq5DEYKOTqVHTsQl4rcwmKxbC57ToiIyIAZSV0BqhsHK1Oc/L8hsDSt+E/Wp40jzt7KQOKDXOQUVASSzLwiKapIRESkFQwnBsS50u7FLjamAIBfTyTg1xMVs3buZRfqtF5ERETaZDCPdagqF1szjcfvZhUgv6gEJ67fRzH34yEiIgPDnhMDVrknpdzdrAJ0XLQbggC8ObgtZj/eXsc1IyIiajj2nBgwJ2tTjcf/d/oWyndM+vFIvA5rRERE9PDYc2LAPFpY4K3BbWFpaoRWDhaY9uvpKmWM5DIJakZERNRwDCcGbpbKI5vLnwxHu/d2qZ3PKiiGIAiQyRhSiIjIMPCxThNiUs0ePAlpuTquCRERUcMxnDQDR67ck7oKREREdcZw0gyEX7sPAMjILcL52xkS14aIiKhmHHPSxEzq4411x2+oHdtxLgn7F+5GXtmy9n9N642e3vYS1I6IiKh27DlpYh7v5KzxeJ7Kfjt7LiTrqjpERET1xnDSxPRp44g/p/VG79YO1ZYxN1bosEZERET1w8c6TVCgtz1+e7UXdp5PQlJ6PhbvjFUvIJNh/fEbuJtVgHeGcvVYIiLSL+w5aaLkchme6OKGYL+qj3nuZxfg/W0X8PWBq7iSkiVB7YiIiKrHcNLEWZlWdI4939MDALAhomIH4yup2dh+9g43CCQiIr3BcNLEOVqZYGB7Jzzu54xne3gCgLjvDgC88dtpTN8QrRZYiIiIpMQxJ02cTCbDuslBAICrqdnVljt46S5e7u2to1oRERFVjz0nzYidhXG15xTcIJCIiPQEw0kzYmdefTjZdzEFBy6lQqkUqi1DRESkCwwnzYiRoub/3JPXRmLHuSQd1YaIiEgzhpNm5seXe9Z4/t/YFB3VhIiISDOGk2ZmQHsn2NbweCchLVeHtSEiIqqK4aSZMVbIEb3wMVz+ZLjG89EJ6RAEjjshIiLpMJw0Q3K5DCZGcpx6LxgWJlX32fGZvxPe83Ygkb0oREQkAYaTZszRyhQXPxqG4Z1dNJ5/b8t5HdeIiIiI4YQAzH68HUw0zOS5nJKFuORMJNxnDwoREemOTDCwAQaZmZmwtbVFRkYGbGxspK5Ok5FbWIy45CyM+ea4eMza1AhZBcUAgBvLRkpVNSIiagLq8/3NnhMCAFiYGKG7Vwu1Y+XBBADWH7+BxLRcvPV7NE4nPNB19YiIqBlhzwmp8Z63o07l2JNCRET1wZ4TanT3sgvwR2QicguLay9MRERUD9yVmNT4trTClRp2Ly7X85N/AQDnbmfg49GdG7taRETUjLDnhNR8/3JPDO/sgn+m98OemY/WWv6XEzfF17mFxdh7IRl5hSWNWUUiImri2HNCanwcLbHmpR71+kxxiRJGCjne/esstp9NwtPd3PHl2K6NU0EiImry2HNCD62gWAkA2H62dEfjv6NvS1kdIiIycAwn9NA6vb8HEfFpUleDiIiaCIYT0ornvwuXugpERNREMJxQjeYMbS++NjdWwMaMw5SIiKhx8ZuGavTGwDZ4qqsbXG3NUVSixPQN0fg3NkXqahERURPGnhOqkUwmg0cLCyjkMpgZK+Dnal2nz+29kIx/Yu6oHSsqUTZGFYmIqIlhOKF6+c/Atnixlxd+mRJUY7mpv0Thzd+jcTerAABw5MpddH5/D+b+dVYX1SQiIgPGcEL1Ym6iwJKn/dHf16lO5Sf8dBJ7LyRjwk8RKChWYtOpRLXzt9PzEH7tfmNUlYiIDBTHnFCjikvOwtRfoqo933fZfgDA//7TGz1a2euqWkREpMfYc0IPrWerFvUqX6KsuhH28avsPSEiolIMJ9Rgnz3TBb187PHjxJ64sWwkAjzt6vS5307erHIsv5j78RARUSmGE2qw5wM9sen13rCzMAEAvNTLq06fW7T1ApSVek8KijiTh4iISjGckNaM7uaO1we0rlPZvCL1npL84hJE3XyA3kvDsPNcUmNUj4iIDATDCWmNsUKO+cM71qlsdkGx2vu8QiVeWReJpIx8vPHb6caoHhERGQjO1iFJ3E7Pw9Er98T3mflFyMgrkrBGRESkLxhOSOse93PG3ovqS9wPau8EhVwuLn0/5pvjaudTyxZrIyIi4mMd0roVY7vi7eB2asde7u2NHyf2rPYzqZn5Go8LgoC0nEKt1o+IiPQbwwlpnZWpEWYE+2LpGH/xmKOVKQDg0XaaV5ZNqRROytdC+Wj7RXT/eB8OXb7bSLUlIiJ9w3BCjWZUgJv42t6qdLrxuEBPjWUrr8s25IuDAIC1x24AAD7dFaf1+hERkX7imBNqNJamRni2hwcEAXCzNQMAOJT1oNTmxv1cCEJFYhEAfH/4Go5cuYcfXu4JUyM5ZDJZY1SbiIgkxnBCjerz5wLU3tuaG9f5swcrPcpZsrO092TS2ghcTsnGmvHd0au1w8NXkoiI9Aof65BO2ZhX5OF+bR1rLDt5baT4WrUX5cT1NKTlFGLar9VvKEhERIZLknCyfft2tG/fHr6+vvjxxx+lqAJJxMasouckZFBbxH08rMHXepBbhD9PJSKvkPvyEBE1JToPJ8XFxZg1axb279+P6OhoLF++HPfvc0fa5sLCRCG+NjOWw8xYUUPpCoXFmvfemfPXWXwVdkUrdSMiIv2g83ASERGBTp06wd3dHVZWVhg+fDj27t2r62qQRGQyGaYPaotRAW4I8LADADzbw6PWz12/l1Ptud3nuRcPEVFTUu9wcvjwYTz55JNwc3ODTCbDli1bqpQJDQ2Ft7c3zMzM0KtXL0RERIjn7ty5A3d3d/G9u7s7bt++3bDak0F6Z2h7rBrXDXJ56WybGUN81c471nFGTznbsl2RiYioaah3OMnJyUFAQABCQ0M1nt+0aRNmzZqF999/H6dPn0ZAQACGDh2K1NTUh64sNU2e9hb4d9ajeGNgG+ya0R9/v9GnXp9XyIDiEiViEtNRXKLEH6cS8eOR61XKZeUXYWNEAh5wxVkiIr1W76nEw4cPx/Dhw6s9v2LFCrz22muYPHkyAODbb7/Fjh078PPPP2PevHlwc3NT6ym5ffs2goKCqr1eQUEBCgoq9l3JzMysb5XJALRtaY13h3Vo0GfvZheg7YJdAIA3B7fF6v1XAQAutmZ4zM8ZGyMS0aeNA74Ku4LtZ5OwOfo2/ni9t9bqTkRE2qXVMSeFhYWIiopCcHBwxQ3kcgQHByM8PBwAEBQUhPPnz+P27dvIzs7Grl27MHTo0GqvuXTpUtja2oo/np6aVxilpmVkF1fYW5rg7AeP11o2MS1PfF0eTABg+oZo9Pj4X7y/7QIe+/Iwtp8tHZsSEZ8mlsktLMbaY/G49SBXi7UnIqKHodVwcu/ePZSUlMDZ2VntuLOzM5KTkwEARkZG+OKLLzBo0CB07doVs2fPhoND9QtpzZ8/HxkZGeJPYmKiNqtMeurrcd0Q9V4wbMyMEeBp1+DrZBcU13h+2a44fPjPRTz19bEG34OIiLRLknVORo0ahcuXL+Pq1auYOnVqjWVNTU1hY2Oj9kNNn0wmE5en/+8rmh/7PebnrPF4XZy7lQEA4oaC9zkOhYhIb2g1nDg6OkKhUCAlJUXteEpKClxcXLR5K2pGbM2N8cGTfgAA35ZW4vEfXu6Jj0d3btA10/MKUVSixM37fJxDRKRvtBpOTExM0KNHD4SFhYnHlEolwsLC0Ls3ByBSw03q64Mby0ZiU9lAVh9HSwDAS728GnS9vRdSsP74DW1Vj4iItKjes3Wys7Nx9WrFoMP4+HicOXMG9vb28PLywqxZszBx4kT07NkTQUFBWLlyJXJycsTZO0QPw97SBDGLHoeZSWmulslkCPRugcgbD+p1nV9O3KxybFXYFUTeSMOPE3vC1KhuK9cSEZH21TucnDp1CoMGDRLfz5o1CwAwceJErFu3DmPHjsXdu3exaNEiJCcno2vXrti9e3eVQbJEDWVrob6zsQwyrVx3xb7LAIAdZ5MwprsHEtNyMfWXKPi72+CzZwM0fqaoRIk/T91C37YOaOVgqZV6EBE1dzJBdbtXA5CZmQlbW1tkZGRwcCwBAJ7/LlycHhz6YneEbDj9UNd7b2RHXEnJxqZTFTPDriweDmNFaW9NUYkSd9Lz0MrBEt8duoalu+IAADeWjXyo+xIRNWX1+f6WZLYOkTa9N7IjZDJg+qC2GNnF9aGvd/1ejlowAYDkjHzx9dz/ncWA5Qdx4FIqjl3jppVERNrGcEIGr4uHHWI/GoZ3hrbXyvU2nEyocqz/ZwfE15tPl65wvHRnLJRKg+p4JCIyCAYTTkJDQ+Hn54fAwECpq0J6yMy4YgDr4qfVpxd72ptr5R6VV5FNSs9HCcMJEZHWccwJNUklSgHJmfmIiL8PO3MTTF4XqXbeRCFHYYmyXtfcM/NRtHexhve8HQAAUyM5jBVycRXa2sacpOcWwo47KBNRM8UxJ9TsKeQyuNuZ4+luHhjY3gnvjeyIX6ZUrDTrYmuGSX2863XNvKIStfdFJUqNy+NfvJOJ1385hSspWeKxzadvoetH+xB64GqV8kREpI7hhJo8mUyGV/u3Rn9fJ/GYhYkCH4zqhLeD29X5OjmVgkh1T3TGfh+OPRdSMGltJARBQEZuEWb/GQMAWL7nUv0bQETUzDCcULPy5dgAeLQwx4rnuwIAZgT74uri4XCxMav1szkFxSgsrv1RUFZ+aYi5nZ6H2X/GIOCjvTCsh6dERNKq9yJsRIbs6W4eeLqbh9oxI4UcyjqkhwOX7mL6huhqz/8ekYCnu7mrHSuf2UNERHXHnhMiADVFE1Oj0v+b/B6RUOMg2vmbz2H2HzG13mvQ5wdx8FJqfatIRNRsMJwQAdA0aW1UgBuiFz6G6YPa1vk6O84l1Vom/l4OJq2NrLUcEVFzxXBCBODNwb4AAHe7ijVRWlqbooWlCfr5OkpVLQiCgPO3M5BfaaYQEVFTxjEnRABe7t0KvVrbo62TFX47mYCwuFRM6usNAGjtaNXo94+6mYbYpCyM7+UFmaxiI8O/o29j1h8xeKS1PTZO7d3o9SAi0gcMJ0QonW7cwaV0UaCJfbwxUWUNFCsz7f/fRC4DrqZmY8nOWMwM9sUza8IBAK0cLNCvrSN+PBIPPzcb/Fa2lP6J62niZzPyijBjYzRGd3XH6EoDcImImgKDCSehoaEIDQ1FSQm7t0m3FPKKnowXe3nhsY7OVVacrS+ZTIbgFYcAAPvjKgbHvvNnDIZ0dBb39+nZqkWVz4YeuIqDl+7i4KW7DCdE1CQZzJiTkJAQXLx4EZGRHEhI0rE2NcKgDi2xZnz3h7pOdXvypGQWqG08eOrmA/F1+biTPyvtmExE1NQYTDghkpKDZemeOI93cgEA9K00SPbTZ/zF1xMeadUoddhwMgHnbmXgQW5Ro1yfiEhfMJwQ1cGetx/FlpC+6FH2mMXGzBjfvlTaezJ3WAeMDfQSy/bQ8ChGG3IKinH4yt1GuTYRkT7hrsREWhKTmI5TNx9gch9vtP6/nRrLPNfDA39G3dLaPSvvhJyeW4gXvj8BOwtj/DqlF4wUpf/+2HMhGZ/uisPKF7qii4ed1u5PRFRX3JWYSAIBnnaY0s8HcpUBtJUtGNkRowLctHbPqJtpWL4nDoXFSgiCgGNX7yMuOQsnrqfhdnqeWO71X6JwnYu/EZGBYDghagRPdHEVX48LqnjkY2qkwOzH674Tcm2eWROO0APXMOevGPRdth8hG06L59Yeu1GlfFpOITLzixC84hA+3R2ntXoQEWmTwUwlJjIkX73QDW8N8YWxQg4fR0s86usImQwwN1GglYMl/jOwDdYcvKa1+209c6fKsXXHb2DhE35qU6EB4LcTCbiamo2rqdmYO6xDtddUKgUcuXoPXdxt0aJsQDARkS6w54SoESjkMrRztoaPoyUAYLi/K4Z1ruhNUc0L7Z2tG60eey8kAwBUFp1FdkHV2T5Z+UX49tA1JKblisc2RiZi4s8RGP3NsUarHxGRJgwnRBLo19YJQGlo2PP2o412n4y80iBipJKGQg9U9Nj0WvIvDl++i0+2x2LZrjg8/c1x8dz2s6W9MTfvVwQWIiJd4GMdIgn0buOAjVMfEXtWKrM1NxaDxcOwMTcGABjJ5SjSsLpySmYBXv45Aq62ZgCAe9kF2HcxBcv3xCE1q+Ch709E1BDsOSGSyCOtHeBsUxoKfny5JzxamOOjpzrh+Z4e2DWjP57o4opxQZ4PdY8SpQBBEJBXj12NX/vvKVxOyUY6F3sjIomw54RIDwT7OSPYz1nt2Ncvli7y9ntEw5erzy8qwfzN52otV9tqR1/uu4yRXVzRrhHHxxARlTOYnpPQ0FD4+fkhMDBQ6qoQ6ZSxonS8iExWuujalcXD6/zZX08mYGNk7eEmOTO/xvNfhV3BE6uP1vm+ldd2VCoFvL/1PP7S4gJ0RNR0GUw44cZ/1FxtnNobAR62+GtabwCAsUKOvXUcRBuTmK61ehQWK9XeJ6blYuuZ2+ImhoXFShSXKHE3qwBBS8LwyfaLYtn9calYH34T7/wZo7X6EFHTxcc6RHquR6sW2Dq9n9qxds7WGBXghm0xVdc30YWkjDz0/+wAAGDGxjNYMKIjfjx6HS0sTDCoQ0vczSrAj0fj8d4TfgCABVtqf7RERFTOYHpOiEidUsNAEXtLE6ybHIjW1cwCeliXU7LQZ2kYei/dr3Z88c5YpGQWIC45CwVFFT0sd7MKkJFbhJRMzvwhorpjOCEyUJoGsQZ3bImB7Vti/StB4rEWFsbwtDdXW4itoR7/8jDuZNQ8PuXf2BTx9ZKdsfWaKUREBDCcEBksX2cr8fX2N/vhiS6umNzXBwDgaW8hnuvkZosj7w7G8mcDdFKvBJVVZlOz8pFTWKyV60beSENccqZWrkVE+o1jTogM1LQBbZBfpMTjnZzR2d1WnHpcWXmPSSe3ii3KWztaopO7Lf5p5DErtx7kYdamM2rHBEGArKxSpxMeIDohHbFJmfgr6hb+b0QHTH20TZXrJGXk4blvwwGUzliqier1icgwMZwQGSgzYwXmDa9+477KOrra4NuXesDazAh92zoCAF7t54OXf47Qymq0mty8n4ublY4VKwXkFBTBzsIEY1SWyweAJTvjNIaT63dz6nS/9NxCjFx1FCP8XbBgpF9Dq01EEuNjHaJmZFhnFzGYAECApx1i3n8c8UtH6KwOvgt2oetH+2pd8yS3sBhbom8jI69IbRqz6hoqGXlFau/XH7+J2+l5+OFIvPYrTkQ6w3BC1MRZmdbeQVrTY5BvxnfHtAFVezMeVm1rnqzefxUzN53Ba+tP4ciVe+Lx4rJ1Vc7fzkDAh3sx+48YlXPKKtchIsPDcELURH32TBd0cLHGgpEd61TewdIEADAqwA3h8weLx40VcvTysW+UOtZk17kkAEDEjTT8fKyiJ6R80bc1h0p3V94cfRtxyZl48YcTiLyRpvN6EpH2MZwQNVHPB3pi98xH4dHCovbCAP55sx+mDWiDj57qBFdbc7VzA9s7YUx3d/H9mvGaB99qQ3TCAxSXKKutd8hvpxF1Mw0lJRWPc6asO4Xj1+7jxHWGE6KmgANiiQgA4GZnrnGArb+7LWQyGT57pgvupOehbUsrDPd3bbR6bD59G6/9Nwr3sjUv3BYWl4qwuFQEd2wpHruTkdcodUnKyINCLkNLa7NGuT4RacZwQkQaRb0XjKz8YrjYln4xGynk2Di1t3i+s7sNzt/W/rojW8/cRmZ+7WujlI89AQCFTIbiSqvSrfz3Mnq3dkCv1g5VPnvxTibu5xSgv6+TeOxOeh4sTY1ga24MoHRAbvlKuNeXjIBczunJRLpiMI91uCsxkW45WJnCu4Zl8LNVAkSQtz1+f+0Rrdy3LsEEAOLvVUwvVg0q5Vb+ewVjvz9R5bggCBix6ggm/BSBWw9KF4xLzcpHn2X70fOTfWI51SX3C0s40JZIlwym5yQkJAQhISHIzMyEra2t1NUhavayCypCxB9lOya/NcQXCfdzkJVfjLC4VADAzrf645MdF3H82n2t3v/m/dzaC1USFpuC97ddUHmfihaWJggrW3K/SGUcy6Kt58XXmsIPETUegwknRKRf5g/viNl/xmBKPx/x2KzH2gEAUjLz8dH2i5jY2xt+bjbY8NojKFEK+O7wNXy2+5JO65mckY9/Yu7AzsIYc/46q3ZONaiUK1EKUMhlatOXi4qVgGmjV5WIyjCcEFGDPNPDA33bOsLZpuq3trONGUIrLaevkMtgYazQeK03B7cFULq2ibYN/uIgcgvrvvlgXlEJLE3U61nUwMc628/eQXZ+MV4I8mrQ54maK4YTImqw8sGydWWkqBjm5mhlKs7Imf14e2TmFzVKOKlPMAGAfReT4dvSWu2Y6piT0wkPsDX6Nq7fy8GRK/fw7UvdMaxz1dlLJUoB0zdEAwAGd2iJljac8UNUVwYzIJaIDN/T3dzR0dUGrw9ojVUvdAVQOk4FAGzMjNHSWr0Xxtuhbmu0aNPbm2LwxOqjasci4tPw1NdH8e/FFIz55jjWh98UH/tM+/W0WtmcgmIIgqA2YLigmANqieqDPSdEpDOWpkbYNaO/+P7cB4+rLa+vuor+zrf6o5WDBZ79NhyxSdqfslwfa4/dwLnbGXj1v6c0ni/fCflMYjqeWXMcr/bzwUuPtNJxLYmaDvacEJFkrM2M1fb1MTGq+JXk52YDS1MjbJvet9rP21kYN2r9yp27nVHj+ZyyR0df7L1UNvD3Os6rfObApVSkZuUDAE5cv4/EtFzcSc/D89+FY/f5pMarOJGBYjghIr3x9bjusDRR4FWVGUDGCvVfU53cbAAAn4zuLO6zI7W9F5IBADbmFWFp8c5Y8fWirRfw2IrDOH87Ay98fwL9PzuAD7ZdQER8WpXHQkTExzpEpEcCPO0QtfAxmFWa1XNozkCcvJ6GYqWAcUGeSMksgIutGZbtilP7bExiuo5rXGrLmTsY090DNmYV4aSVgwVuPahYVj8jrwinEx6I7x/kFoqv/ziViJbWphjYvmJJfqLmjOGEiPRK5WACAK0cLNHKoWK12vJZQvaWJuJicOsnB6LrR6UrvHbxsEV2QTGu382pcq3qKOSyBvfEtHa0xKXkLChVPm9qVLUdquflKo+z3i1bf2XBiI7wtLfAsM4udbpvUYkSgqD+OIyoKWA4ISKDFfpid8z+8wzmDusAOwsTbJr6CMKv38ebg32hkMuQnJGPR5aGAQC+eqErZmw8U+21HiacrDt+A+uO31A7lpyRX6WcygK0GledLX8UdGPZSI33KSpRYvX+q1gVdgXzh3fA7xEJKFYKODRnEBRle/8s3xOH+Hs5+Hpcd+4HRAaL4YSIDJa/hy32vj1AfN+r0kZ/pio9Ct29WmB8Ly/8djJB47UEQbvjV8oHwFZ3j6ibD6qcr8n8zeew4+wdce+hpSqPtBLTclGsFNC2pRVCD1wDAJzq8wBBPvYNqTqR5BhOiKjJqvy4I6+oYkG2Vg4WsDEzFmfi2JqbiIvCqRrh74Kd55Lrfe972YVVjn2yI1ZDyaqKSpT4X9QtAMBfUbeQmlWAhLTq9xJ6+ptjeJBbhMl9vcVjhVxbhQwYwwkRNVmq41ccrExQUFTxhR02awBkMhna/N9OAMAr/bwRk5iOJ7q44eytdPxwJB4j/V3x9mPtNIaT0V3d0NrJCiv2XdZ6vV/84QQib9S9Z+VBbhGA0vVYqpORW4SPtl9Er9b2eL6n58NWkahRMZwQUZOlkMtwbN5glJQIsDAxQkdXa+w4V7quiFGlKcomCjm+m9ATADC0kwv6tHFEkI89kjOrPp4BgJUvdIMgCI0STuoTTOpq/t9nsfNcMv53+hbaO1sjwNNO6/cg0hYO8SaiJs3dzhxeZcvgv9q/NeYMba+2Sm051f1zTIzkGNShJSxNjdDa0RKP+TnD3tKkymdkMhle7m0YK8Huj0sVX7/9xxm1c+duZSD82n3cvJ8D73k70Pb/dqKguOIR2NXUbCSm5SIjtwjDVh5G6AHt74FEpMpgek5CQ0MRGhqKkpL6beJFRFTOzFiBkEFtNZ7r6mGn8bhMJsMPL5f2qCzaeh7/Db8Jdztz8by+LARXWWFJCcZ8cww+jlbwdbZCvsojrTvp6uuvPPl16V5CHV1LF7grVgrYfT4ZT3V1R0ZeEYJXHAIAzHqsHeKSsxCXfAmv9veBIGie+k30sAwmnISEhCAkJASZmZmwtbWVujpE1EQcmjMQ1+5mo09bx1rLzgxuB48W5hjcoWKxNAuTii/nKf18EOjdosZVX1tamyI1q+rAW23bEn0HpxPScTohvcq5/CIlwmJTMKSjM87dqlhm/1JyxR5G4dfuo4uHHfJVBhHnFFRsZjhw+UFkFxQj6r3HuM4KaR3/RhFRs9bKwRKDOzjXqay9pQmmPtoGbVtai8f+M7AtHvNzxnsjO2LhE34Y1tkVX73QFTZmRgjwsIW/u/o/pjStb9IYtsXcqfH8lPWnkF9UojaDSbVqGyMTMejzg2o9Q6qPvpIy8pGVX4xbD6qfRaTJ1dQsHLlyt16foebHYHpOiIj0kb2lifjYp9xTXd3xVFd38f3ZW+mYviEac4d1wLzNZ6u9lrWZEbLyi6s9r22Bi/9V6xnRRDW8FJVUnZ6sKWul5RRiyc5YeLQwx7W7OfjgST9YmhrBzFiB4BWHAQB7334U7Zytq364Aa6mZiP8+n2MC/SsMtCZDBPDCRFRI+viYYfD7w4CAHzwzwWNASTA0w5bQ/oiu6AYnd/fo5N6Va6HpnCUrfIop6i4ahJ5b8s5BHjaYd6wDuIO0x9su6DWc/NPzB0YyWXYNr2feOxqajbaOVsjNSsf4dfuY3hn1wY/HiofEwNBwITe3g26BukXRkwiIh0KfbE7jOQyBHjYYoS/C4aX7aPz7tD2AAArU+n+zagpNGWrHNt0KrHK+RPX0/Ddoeu49SAPf0Qm4tPdcbicklWlXLFSwPI9Fava7r2QjCFfHETQ4jDM2HgG7205p7FO529nYMw3x3Di+v1a6x8t0caPpH3sOSEi0qEgH3tELXwM1qZGkMtlEAQBGXlFsLOomKr831eCcPTqPXx/+HqN17I1N0ZGXpF43Yj4NK3XV1PQ0OSXEzdrra/qI6AtZ9THxPxx6hYm9fGBn1vpjKGb93OweEcs9l5MAQC89ONJXF0yosbrG8v57+2mguGEiEjHbM2NxdcymUwtmADAo+2c8Gg7J/xnQBtcuJOJl346CaB0zZZpA9tg4ZbzAIBA7xbo0coe7ZytMKSjMy7eycSIVUe0WtfV++u2pkltwQQADl2ueSDsjnN3xHAy6utjYvAC6jaQeNOpRHg5WFQ7XZwMB2MmEZGeamFpgvYuFYNGf3/tEUx4pGLRt2KlgP8MbIMhHUtnG/m52aBfDVOi3e3M4aBhMTl9obofkGowqY/ley6hsFiJD/+5gFfXn4JSdbZRsRIPcqrueUT6h+GEiEiPOVmbYvmzXTBnaHtxpVsv+9L/LR+vomrpGH8EeNhi1bhuMDNW/xV/5N1BCB3fvfEr3UC/RyTi1oNctUChKqZsTElxiRIv/nACC7ec17gIXrFSibXHbuDf2BS1cShPrj6Kbh/vQ1JGXpXPkH7hYx0iIj33XKWN+raG9MX5Oxno26ZqL4mnvQW2ls2Keev3aPH4jrf6QS6XwcbMWK18JzcbXLiTCX2QXVCMfp8eqPZ85I00BHja4VJKFo5fu4/j1+6Lq9qqyi2smP5coDIV+lLZ+Jmw2FS89IhhbDvQXLHnhIjIwLSwNEF/XyfI5bIayzlamYr/28mtdDG4Di7WaK+yvsh/XwnCsjH+jVdZLUpIK13wrbikorfkz6iqM4hUZx1pGquSX1SC3MK6rycTeuAqBi4/gLsqK/tm5BXh+NV71fby0MNhOCEiaqLWTgrE4A4tseG1XuIxuVyG3TP74/0n/fDZM13gYGWKF4K8JKxl3SWWhZMclWChun1AuUyV8Srl41gEoSJEfLIjFr0WhyEjt7RcUYkSu88n4352Rfi4lJyFBX+fQ3JGPpbvuYQb93Px3aFr4vln1hzHiz+exB8aplfvOJtU51lOpBkf6xARNVH+Hrb4eVJgleMymQyT+/o0yj1f7t0K/w2/2SjXPnDpLq6mZiG3oOJRzZ30/CrlMvMrwklYXCqW7IpFa0dLtTJZBcXYfu4OhnRwxv9O38LyPZfQtqUV1k4KxIaIBKw5WBpEyntrAPVemKup2QBKtwlQDXd/Rd3CO3/GAABuLBv5MM1t1hhOiIgI618JwofbLmDZM12w81wS1h2/gX5tHXH06r06X+N//+lTp8XSHsabv59Rm3ETfy+nSpkJP0WIr3+PSAAAXL9btdyCv89jAc6L76+mZqP/Z+pjXuKSK3pAZLLSHpW2La3UjpWLvJEmBpOaCIKA0ANX0cnNFoNUNpGkCgwnRESEAe2csP+dgQBKF3T7YFQnAKW9EBtOJmDZrrhqPzummzvatLRCdy87RCc8qPE+liYK5BTWvJ9PTWKTdDt4V/Wx0frjN7D22A218zLI1M6r2nrmttoeS2uPxaO4REB7F2t8vvcyAPXelZjEdJy7nYHxvbzErQCaK4YTIiKqlo2ZMar7muzgYo0tIX1hZlzxBf5CkBeW7orTOMUXAIpqGED68ejO+OrfK7inMvZDagVFFWuv1Db21dnGTO39jI1nMCrADTKZDPlFJfjwn4sAgNmPtRPLpGbmI7ugGK2drPBU6DEAgIuNGYL9KnbKFgSh2YUVDoglIqIaqX4p/zvrUex7+1GM7emJH17uqRZMgNK9gXa8VbHB3+N+zhjhX7Eei+pCawuf8FP77Eu9vPBUVzct1/7hJGdWHdOiSiYDDlxKxY9HrsNSw+Dc8kdMquNgbj2oWGclaEkYBn9xSG0m0OXUikdJq8OuoPfS/bidXr+1Wc4kpmP+5nNqg3wNCcMJERHVSKky06VtS2v4Olvj02e7wLNsMbjKjFSmOC8Y2RHfjO8hvnewNEGQjz0GtHPCxN6t0NqpYqCqTCbDxDruKjy5b93KNbaMvCJMXhuJT3bE4nJKdpXzR6/eQ2xSJs7dyhCP3dGwCJzq2BmFSi/JF/suIzkzH6EH6raNQLnRocfwe0QCPijrrTE0DCdERFSj8pVoO6gspV+zii9X60qLvhUUK7Fp6iNY/0oQjBRy7J7xKPxcbTC0U+ljDC8HC3w3oQcqezu44lHI0bmDsHCkX5UyUkhUmc2z+0KyxjLDvzqCKetPie+PXKk6yFh1NV+FhvVrilR6nOrjcrJhTmlmOCEiohq1drLCyf8bgq3T+9apfL7KqqxWpqVDGyf18QYAvDusvdr4CRMjOXa81Q/fTegpHnOyNq1yzSAfewClj1E8WlhALpfB3c683m3Rtge5DdsDqLLsgoq1W+5mF6itywIAf0bdwqqwKw267oq9l3CllnVX8otKcOTKXbX/dlIymHASGhoKPz8/BAZWnbNPRESNy9nGDKZGVcdUVFe2nIlR6dfMoif8cGjOQLWNC8tVHuxZOXQo5DL08rHHpqmPIHzeEPH4yhe6wtrUCMvG+OOT0Z0xLsgL/84aUOc26ZMXfzgpvv7u0HV8uvtSlaCwYt9lXLtb8ehIEATczy7AjXs5mLQ2ApE30qpc93Z6Hlbtv4rHvjyMXkv+1bhoHAB8tP0iJvwUIe54LTWZUDme6bnMzEzY2toiIyMDNjZV91QgIiLpRd18ABszI/g61/VRkLplu+KgFAS8NcQXchlgYaJ5cqlSKVRZxv9Oeh4ib6RhxsYzGj8zprs7Np++Lb4f3dUNW87caVA9G1MbJ0tcq7Q+y3cTemBop9LHbLP+OKPWDgA498HjUMhl8Fu0p9rrrp0ciEHtK9ZXEQQBPvN3iu8ba/G4+nx/cyoxERFpXY9WLR7q8/OGd6hTOU37C7nZmcOjRfWPfPJU1ln5vxEdMOERbwxo74S3N9W+gJouVQ4mQGndS5QCSpRClWACAP4f7K31upPXRqoFkMnrItXOhx64ipBBbRtQY+0xmMc6REREdVWiMn7U390W0we1RUtrU6wc2xVPdytdGK2Lhy2mPtoG5iYK9G5ddYfnmjwZUPuU566edvW6Zl28+7+zaPN/O9HuvV1aud7drAIcvHRX7djyPZe0cu2HwZ4TIiJqcjq6lj5OamFhjH/eLF13Zfbj7SCTySAIAra/2Q8+KvvtGCtqXuRsTHd3XE7JwvnbpSvUetnXPhh3RrAvdpxNwl9RtwAAvVs7IPwhl/cvbOCsneqcvZWu8bjUC78xnBARUZNjbWaMM4seUxvEW/5lK5PJ0NndVq28kaLiQcLT3dxxOuEBbt6vmCa84vmuAICd55JwL7sA3TxbIPTANdRELpOJs5UA4Pepj+DCnQyExabC1twY72+70OD2PazBnx/EdQ37EpUrVgq1BrbGxHBCRERNkp2FSZ3Lqn4RP9/TE1+O7QrveTuqlBvh7yq+Xv5sF7jZmWP72SRxg0EzYznyy5a892xhLs5WKtfJzRad3EqDkZThpKZgApSuR2OskG7kB8ecEBFRs6f6RWxatiCab9nuwxrG3AIAnuvpib5tHZFXWLFGSfTCx/H3G33w/YQeaO1khdf6t4a7nTneGlx1gOl7IzvCz9UGx+cN1mJLtKNA4vVO2HNCRETNnuqS+y3Kely+f7knFu+IRcigNjV+dkJvb2w5cweDO7SEuYkC3bwqZio5WZvi6NxBGsdvvNq/NV7t3xoAcH3JCOw8n4TpG6K10ZyHlq/lsS31xXBCRETNnkwmw6In/JCRVyQOlPVxtMSPE3vW8snSadMn5g+Bo5Xmx0h1GVgql8vgZFV1ZVypLN5xUW1PJF1jOCEiIgLwSj+fBn/Wxdas9kK1sLUwrr2QjphION4E4JgTIiIivWBrXjWcfDO+Oz57tgsWP90ZP7xcey+Oqg+eLN0c0cJEgYsfDa3XZ10l3reIPSdERER6QFM4UZ0dVNn84R2wdFdctecn9fXBuF5eKFEK1S7/r4mjlQmmPEQvkjaw54SIiEgPmBsr0NHVBg6WJgju2BJfv9itxvKvD6gYqNunjQPMjatuzGhqpBCDybvD2qO/b80r4b41xBcn/y8YjhKPf2HPCRERkR6QyWTY/mY/KAWhzmuMrBzbFVE3H+CDUZ0AAON/PIET16vuTgwAbwxsizcGtsW0X6Kw+0Iy+vs64siVe+qFBAGK6uZO6xB7ToiIiPSEQi6rMZj8OqUXPFqY49cpvQAAo7u54+PRnaGQy8Sf2nwwqhMWPeGncQzLsz08G155LWLPCRERkYHo5+uIo3OrX7StTxtHHLt6HzXNXnaxNRNnJjlameJedgE2vNoL/h62sDbTjxlDDCdERERNxGv9W8PW3LjWsSXlwmYNwI37OQhohB2UHwbDCRERURNhYiTHS4+0qnN5WwtjBFjYNV6FGohjToiIiEivMJwQERGRXmE4ISIiIr3CcEJERER6heGEiIiI9ArDCREREekVhhMiIiLSKwwnREREpFcMJpyEhobCz88PgYGBUleFiIiIGpFMEARB6krUR2ZmJmxtbZGRkQEbGxupq0NERER1UJ/vb4PpOSEiIqLmgeGEiIiI9ArDCREREekVg9uVuHyITGZmpsQ1ISIioroq/96uy1BXgwsnWVlZAABPT0+Ja0JERET1lZWVBVtb2xrLGNxsHaVSiTt37sDa2hoymUyr187MzISnpycSExOb5Ewgts/wNfU2NvX2AU2/jWyf4WusNgqCgKysLLi5uUEur3lUicH1nMjlcnh4eDTqPWxsbJrsXzqA7WsKmnobm3r7gKbfRrbP8DVGG2vrMSnHAbFERESkVxhOiIiISK8wnKgwNTXF+++/D1NTU6mr0ijYPsPX1NvY1NsHNP02sn2GTx/aaHADYomIiKhpY88JERER6RWGEyIiItIrDCdERESkVxhOiIiISK8wnJQJDQ2Ft7c3zMzM0KtXL0REREhdpTpZunQpAgMDYW1tjZYtW2L06NG4dOmSWpn8/HyEhITAwcEBVlZWeOaZZ5CSkqJWJiEhASNHjoSFhQVatmyJOXPmoLi4WJdNqZNly5ZBJpNh5syZ4rGm0L7bt2/jpZdegoODA8zNzeHv749Tp06J5wVBwKJFi+Dq6gpzc3MEBwfjypUratdIS0vD+PHjYWNjAzs7O0yZMgXZ2dm6bkoVJSUlWLhwIXx8fGBubo42bdrg448/Vttfw9Dad/jwYTz55JNwc3ODTCbDli1b1M5rqz1nz55F//79YWZmBk9PT3z22WeN3TQANbevqKgIc+fOhb+/PywtLeHm5oaXX34Zd+7cUbuGobavsmnTpkEmk2HlypVqx/W5fUDd2hgbG4tRo0bB1tYWlpaWCAwMREJCgnhe0t+tAgkbN24UTExMhJ9//lm4cOGC8Nprrwl2dnZCSkqK1FWr1dChQ4W1a9cK58+fF86cOSOMGDFC8PLyErKzs8Uy06ZNEzw9PYWwsDDh1KlTwiOPPCL06dNHPF9cXCx07txZCA4OFqKjo4WdO3cKjo6Owvz586VoUrUiIiIEb29voUuXLsKMGTPE44bevrS0NKFVq1bCpEmThJMnTwrXr18X9uzZI1y9elUss2zZMsHW1lbYsmWLEBMTI4waNUrw8fER8vLyxDLDhg0TAgIChBMnTghHjhwR2rZtK4wbN06KJqlZvHix4ODgIGzfvl2Ij48X/vzzT8HKykr46quvxDKG1r6dO3cKCxYsEDZv3iwAEP7++2+189poT0ZGhuDs7CyMHz9eOH/+vPD7778L5ubmwnfffSdp+9LT04Xg4GBh06ZNQlxcnBAeHi4EBQUJPXr0ULuGobZP1ebNm4WAgADBzc1N+PLLL9XO6XP7BKH2Nl69elWwt7cX5syZI5w+fVq4evWqsHXrVrXvPSl/tzKcCIIQFBQkhISEiO9LSkoENzc3YenSpRLWqmFSU1MFAMKhQ4cEQSj9RWJsbCz8+eefYpnY2FgBgBAeHi4IQulfYrlcLiQnJ4tl1qxZI9jY2AgFBQW6bUA1srKyBF9fX2Hfvn3CgAEDxHDSFNo3d+5coV+/ftWeVyqVgouLi7B8+XLxWHp6umBqair8/vvvgiAIwsWLFwUAQmRkpFhm165dgkwmE27fvt14la+DkSNHCq+88orasTFjxgjjx48XBMHw21f5F7+22vPNN98ILVq0UPs7OnfuXKF9+/aN3CJ1NX15l4uIiBAACDdv3hQEoWm079atW4K7u7tw/vx5oVWrVmrhxJDaJwia2zh27FjhpZdeqvYzUv9ubfaPdQoLCxEVFYXg4GDxmFwuR3BwMMLDwyWsWcNkZGQAAOzt7QEAUVFRKCoqUmtfhw4d4OXlJbYvPDwc/v7+cHZ2FssMHToUmZmZuHDhgg5rX72QkBCMHDlSrR1A02jftm3b0LNnTzz33HNo2bIlunXrhh9++EE8Hx8fj+TkZLU22traolevXmpttLOzQ8+ePcUywcHBkMvlOHnypO4ao0GfPn0QFhaGy5cvAwBiYmJw9OhRDB8+HIDht68ybbUnPDwcjz76KExMTMQyQ4cOxaVLl/DgwQMdtaZuMjIyIJPJYGdnB8Dw26dUKjFhwgTMmTMHnTp1qnK+KbRvx44daNeuHYYOHYqWLVuiV69eao9+pP7d2uzDyb1791BSUqL2hwsAzs7OSE5OlqhWDaNUKjFz5kz07dsXnTt3BgAkJyfDxMRE/KVRTrV9ycnJGttffk5qGzduxOnTp7F06dIq55pC+65fv441a9bA19cXe/bswX/+8x+89dZbWL9+PYCKOtb0dzQ5ORktW7ZUO29kZAR7e3vJ2zhv3jy88MIL6NChA4yNjdGtWzfMnDkT48ePB2D47atMW+3R97+35fLz8zF37lyMGzdO3CTO0Nv36aefwsjICG+99ZbG84bevtTUVGRnZ2PZsmUYNmwY9u7di6effhpjxozBoUOHxDpK+bvV4HYlpuqFhITg/PnzOHr0qNRV0ZrExETMmDED+/btg5mZmdTVaRRKpRI9e/bEkiVLAADdunXD+fPn8e2332LixIkS1+7h/fHHH/jtt9+wYcMGdOrUCWfOnMHMmTPh5ubWJNrXnBUVFeH555+HIAhYs2aN1NXRiqioKHz11Vc4ffo0ZDKZ1NVpFEqlEgDw1FNP4e233wYAdO3aFcePH8e3336LAQMGSFk9AOw5gaOjIxQKRZURyCkpKXBxcZGoVvU3ffp0bN++HQcOHICHh4d43MXFBYWFhUhPT1crr9o+FxcXje0vPyelqKgopKamonv37jAyMoKRkREOHTqEVatWwcjICM7OzgbdPgBwdXWFn5+f2rGOHTuKo+bL61jT31EXFxekpqaqnS8uLkZaWprkbZwzZ47Ye+Lv748JEybg7bffFnvCDL19lWmrPfr+97Y8mNy8eRP79u0Te00Aw27fkSNHkJqaCi8vL/F3zs2bNzF79mx4e3uL9TPU9gGl33tGRka1/t6R8ndrsw8nJiYm6NGjB8LCwsRjSqUSYWFh6N27t4Q1qxtBEDB9+nT8/fff2L9/P3x8fNTO9+jRA8bGxmrtu3TpEhISEsT29e7dG+fOnVP7P1v5L5vKf3l1bciQITh37hzOnDkj/vTs2RPjx48XXxty+wCgb9++VaZ/X758Ga1atQIA+Pj4wMXFRa2NmZmZOHnypFob09PTERUVJZbZv38/lEolevXqpYNWVC83NxdyufqvGoVCIf7rzdDbV5m22tO7d28cPnwYRUVFYpl9+/ahffv2aNGihY5ao1l5MLly5Qr+/fdfODg4qJ035PZNmDABZ8+eVfud4+bmhjlz5mDPnj0ADLt9QOn3XmBgYI2/dyT/7nio4bRNxMaNGwVTU1Nh3bp1wsWLF4WpU6cKdnZ2aiOQ9dV//vMfwdbWVjh48KCQlJQk/uTm5oplpk2bJnh5eQn79+8XTp06JfTu3Vvo3bu3eL58Otjjjz8unDlzRti9e7fg5OSkN1NtK1OdrSMIht++iIgIwcjISFi8eLFw5coV4bfffhMsLCyEX3/9VSyzbNkywc7OTti6datw9uxZ4amnntI4NbVbt27CyZMnhaNHjwq+vr56MZV44sSJgru7uziVePPmzYKjo6Pw7rvvimUMrX1ZWVlCdHS0EB0dLQAQVqxYIURHR4uzVbTRnvT0dMHZ2VmYMGGCcP78eWHjxo2ChYWFTqai1tS+wsJCYdSoUYKHh4dw5swZtd87qjM0DLV9mlSerSMI+t0+Qai9jZs3bxaMjY2F77//Xrhy5YqwevVqQaFQCEeOHBGvIeXvVoaTMqtXrxa8vLwEExMTISgoSDhx4oTUVaoTABp/1q5dK5bJy8sT3njjDaFFixaChYWF8PTTTwtJSUlq17lx44YwfPhwwdzcXHB0dBRmz54tFBUV6bg1dVM5nDSF9v3zzz9C586dBVNTU6FDhw7C999/r3ZeqVQKCxcuFJydnQVTU1NhyJAhwqVLl9TK3L9/Xxg3bpxgZWUl2NjYCJMnTxaysrJ02QyNMjMzhRkzZgheXl6CmZmZ0Lp1a2HBggVqX2SG1r4DBw5o/P/dxIkTBUHQXntiYmKEfv36CaampoK7u7uwbNkyydsXHx9f7e+dAwcOGHz7NNEUTvS5fYJQtzb+9NNPQtu2bQUzMzMhICBA2LJli9o1pPzdKhMElWUaiYiIiCTW7MecEBERkX5hOCEiIiK9wnBCREREeoXhhIiIiPQKwwkRERHpFYYTIiIi0isMJ0RERKRXGE6IiIhIrzCcEBERkV5hOCEiIiK9wnBCREREeoXhhIiIiPTK/wPKuBgArFk3rgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "plt.plot(train_metrics_history[\"train_loss\"], label=\"Loss value during the training\")\n", + "plt.yscale('log')\n", + "plt.legend();" + ] + }, + { + "cell_type": "markdown", + "id": "66250bf2-3d88-40ad-87e3-7d2b906fd860", + "metadata": {}, + "source": [ + "And eval set Loss and Accuracy - Accuracy does continue to rise, though it's hard-earned progress after about the 5th epoch. Based on the training statistics, it's fair to say the process starts overfitting after roughly that 5th epoch." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "64d54051-358b-4de8-b5b3-04bebf18018f", + "metadata": { + "outputId": "54680efe-bcf5-44df-e468-3054fb1e56bc" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAAHWCAYAAAD6oMSKAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAfRdJREFUeJzt3XlcVOX+B/DPzADDDrKDoqi5kQiKQeRaYqTm72pqaBpGLl2VUrl1lUrRzGi7Xm9eyxY0S72gZotalqJWFoqC5ob7ggszgMgiyAAz5/cHzNERULZhFj7v12teOmeec+Z7wDl+53nO830kgiAIICIiIiKTJzV0AERERETUPJjYEREREZkJJnZEREREZoKJHREREZGZYGJHREREZCaY2BERERGZCSZ2RERERGaCiR0RERGRmWBiR0RERGQmmNiZmEWLFkEikRg6DL2TSCRYtGiRocMwO3v37oVEIsHevXsNHQoREQDg0qVLkEgk+PLLLw0dillgYlcPX375JSQSCQ4dOmToUIhMzp9//olFixahoKDA0KGQgX388ceQSCQIDQ01dCjUyp08eRKLFi3CpUuXDB1Ks2NiR0R69eeff2Lx4sVM7Ajr16+Hn58f0tLScO7cOUOHQ63YyZMnsXjxYiZ2REREjXHx4kX8+eefWLZsGdzd3bF+/XpDh1SnkpISQ4dA1GhM7JrR4cOHMWzYMDg6OsLe3h5DhgzB/v37a7Q7evQoBg0aBBsbG7Rr1w5vv/021qxZA4lE0qhvD5WVlViyZAk6d+4MuVwOPz8/vP7661CpVDrtDh06hIiICLi5ucHGxgYdO3bEiy++qNMmKSkJwcHBcHBwgKOjIwICAvCf//ynzveuqKiAi4sLoqOja7xWVFQEa2trvPrqqwCA8vJyLFy4EMHBwXBycoKdnR0GDBiAPXv2PPAcX3jhBfj5+dXYXtc9h+vWrUNwcDBsbGzg4uKC8ePH48qVKw98H6B+v0ft8Pwff/yB2NhYuLu7w87ODqNHj0Zubm693ufUqVMYO3YsXFxcYG1tjb59++KHH34QXz906BAkEgnWrl1bY9+ff/4ZEokE27ZtAwBcvnwZM2fORLdu3WBjYwNXV1eMGzeu0d9Gi4uLMWfOHPj5+UEul8PDwwNDhw5FRkaGTrsDBw7gqaeegpOTE2xtbTFo0CD88ccf4uuLFi3Ca6+9BgDo2LEjJBJJo/+dk2lbv3492rRpgxEjRmDs2LF1JnYFBQWYO3eu+G+vXbt2iIqKQl5entimrKwMixYtQteuXWFtbQ1vb28888wzOH/+PIC67yWt7V6uF154Afb29jh//jyGDx8OBwcHTJw4EQDw+++/Y9y4cWjfvj3kcjl8fX0xd+5c3L59u0bcp06dwrPPPgt3d3fY2NigW7dueOONNwAAe/bsgUQiwbfffltjvw0bNkAikSA1NfW+P78LFy5g3LhxcHFxga2tLR599FFs375dp432vDdu3IilS5eiXbt2sLa2xpAhQ+rdQ3rt2jW8+OKL8PT0hFwux8MPP4zVq1eLryuVSlhYWGDx4sU19j19+jQkEgn++9//AgDy8/Px6quvIiAgAPb29nB0dMSwYcPw119/1SuWe1VUVGDx4sXo0qULrK2t4erqiv79+2Pnzp067R50bf3yyy8xbtw4AMDjjz8uXpfM5d5jC0MHYC5OnDiBAQMGwNHREf/85z9haWmJTz/9FIMHD8avv/4q3lNy7do18R9SXFwc7Ozs8MUXX0Aulzf6vadOnYq1a9di7Nix+Mc//oEDBw4gISEBmZmZ4oUkJycHTz75JNzd3TF//nw4Ozvj0qVL2LJli3icnTt3YsKECRgyZAjee+89AEBmZib++OMPzJ49u9b3trS0xOjRo7FlyxZ8+umnsLKyEl/77rvvoFKpMH78eABVid4XX3yBCRMmYNq0aSguLkZiYiIiIiKQlpaGoKCgRv8M7rZ06VIsWLAAzz77LKZOnYrc3FysWLECAwcOxOHDh+Hs7FznvvX9PWq9/PLLaNOmDeLj43Hp0iUsX74cMTExSE5Ovm+MJ06cQL9+/dC2bVvMnz8fdnZ22LhxI0aNGoVvvvkGo0ePRt++fdGpUyds3LgRkydP1tk/OTkZbdq0QUREBADg4MGD+PPPPzF+/Hi0a9cOly5dwieffILBgwfj5MmTsLW1bdDP8O9//zs2b96MmJgY+Pv748aNG9i3bx8yMzPRp08fAMDu3bsxbNgwBAcHIz4+HlKpFGvWrMETTzyB33//HSEhIXjmmWdw5swZ/O9//8O///1vuLm5AQDc3d0bFA+ZvvXr1+OZZ56BlZUVJkyYgE8++QQHDx7EI488Ira5desWBgwYgMzMTLz44ovo06cP8vLy8MMPP+Dq1atwc3ODWq3G008/jZSUFIwfPx6zZ89GcXExdu7ciePHj6Nz584Njq2yshIRERHo378/PvzwQ/HzsmnTJpSWlmLGjBlwdXVFWloaVqxYgatXr2LTpk3i/kePHsWAAQNgaWmJ6dOnw8/PD+fPn8fWrVuxdOlSDB48GL6+vli/fj1Gjx5d4+fSuXNnhIWF1RmfUqnEY489htLSUrzyyitwdXXF2rVr8X//93/YvHlzjWO+++67kEqlePXVV1FYWIj3338fEydOxIEDB+77c1AqlXj00UchkUgQExMDd3d3/PTTT5gyZQqKioowZ84ceHp6YtCgQdi4cSPi4+N19k9OToZMJhOTpgsXLuC7777DuHHj0LFjRyiVSnz66acYNGgQTp48CR8fnwf/cu6yaNEiJCQkYOrUqQgJCUFRUREOHTqEjIwMDB06FED9rq0DBw7EK6+8go8++givv/46evToAQDinyZPoAdas2aNAEA4ePBgnW1GjRolWFlZCefPnxe3Xb9+XXBwcBAGDhwobnv55ZcFiUQiHD58WNx248YNwcXFRQAgXLx48b6xxMfHC3f/2o4cOSIAEKZOnarT7tVXXxUACLt37xYEQRC+/fbbB57D7NmzBUdHR6GysvK+Mdzr559/FgAIW7du1dk+fPhwoVOnTuLzyspKQaVS6bS5efOm4OnpKbz44os62wEI8fHx4vPJkycLHTp0qPHe9/48Ll26JMhkMmHp0qU67Y4dOyZYWFjU2H6v+v4etf8mwsPDBY1GI26fO3euIJPJhIKCgvu+z5AhQ4SAgAChrKxM3KbRaITHHntM6NKli7gtLi5OsLS0FPLz88VtKpVKcHZ21vmZlZaW1niP1NRUAYDw1Vdfidv27NkjABD27Nlz3/icnJyEWbNm1fm6RqMRunTpIkREROicf2lpqdCxY0dh6NCh4rYPPvigXv+2yXwdOnRIACDs3LlTEISqfz/t2rUTZs+erdNu4cKFAgBhy5YtNY6h/Xe2evVqAYCwbNmyOtvU9e/84sWLAgBhzZo14rbJkycLAIT58+fXOF5tn6uEhARBIpEIly9fFrcNHDhQcHBw0Nl2dzyCUPVZlsvlOteGnJwcwcLCQudaV5s5c+YIAITff/9d3FZcXCx07NhR8PPzE9Rqtc559+jRQ+da+5///EcAIBw7duy+7zNlyhTB29tbyMvL09k+fvx4wcnJSfx5fPrpp7Uez9/fX3jiiSfE52VlZWJsWhcvXhTkcrnw1ltv6Wy79/dSm8DAQGHEiBH3bVPfa+umTZvqdS00RRyKbQZqtRq//PILRo0ahU6dOonbvb298dxzz2Hfvn0oKioCAOzYsQNhYWE6vVMuLi5i139D/fjjjwCA2NhYne3/+Mc/AEDsqtf2Um3btg0VFRW1HsvZ2RklJSU1urUf5IknnoCbm5tOL9XNmzexc+dOREZGittkMpnYo6fRaJCfn4/Kykr07du3xhBfY23ZsgUajQbPPvss8vLyxIeXlxe6dOly32HfhvwetaZPn64zFDxgwACo1Wpcvny5zvfJz8/H7t278eyzz6K4uFiM8caNG4iIiMDZs2dx7do1AEBkZCQqKip0elZ/+eUXFBQU6PxsbWxsxL9XVFTgxo0beOihh+Ds7Nyon62zszMOHDiA69ev1/r6kSNHcPbsWTz33HO4ceOGeA4lJSUYMmQIfvvtN2g0mga/L5mn9evXw9PTE48//jiAqnJGkZGRSEpKglqtFtt98803CAwMrNEDpd1H28bNzQ0vv/xynW0aY8aMGTW23f25KikpQV5eHh577DEIgoDDhw8DAHJzc/Hbb7/hxRdfRPv27euMJyoqCiqVCps3bxa3JScno7KyEpMmTbpvbD/++CNCQkLQv39/cZu9vT2mT5+OS5cu4eTJkzrto6OjdUZPBgwYAKCqB60ugiDgm2++wciRIyEIgs71MyIiAoWFheK15JlnnoGFhYXONf/48eM4efKkznVJLpdDKq1KM9RqNW7cuAF7e3t069at0delEydO4OzZs7W+3pBrqzljYtcMcnNzUVpaim7dutV4rUePHtBoNOL9XZcvX8ZDDz1Uo11t2+rj8uXLkEqlNfb38vKCs7OzmGAMGjQIY8aMweLFi+Hm5oa//e1vWLNmjc59eDNnzkTXrl0xbNgwtGvXDi+++CJ27NjxwBgsLCwwZswYfP/99+LxtmzZgoqKCp0POQCsXbsWvXr1Eu+PcHd3x/bt21FYWNio87/X2bNnIQgCunTpAnd3d51HZmYmcnJy6ty3Ib9HrXsv5G3atAFQldjW5dy5cxAEAQsWLKgRo3ZoQxtnYGAgunfvrnMBTU5OhpubG5544glx2+3bt7Fw4UL4+vpCLpfDzc0N7u7uKCgoaNTP9v3338fx48fh6+uLkJAQLFq0SOc/Be2FdfLkyTXO4YsvvoBKpWq23ymZNrVajaSkJDz++OO4ePEizp07h3PnziE0NBRKpRIpKSli2/Pnz6Nnz573Pd758+fRrVs3WFg0351EFhYWaNeuXY3tWVlZeOGFF+Di4gJ7e3u4u7tj0KBBACD++9Z+Lh4Ud/fu3fHII4/o3Fu4fv16PProow+8/l++fLnO65L29bs15rqUm5uLgoICfPbZZzU+09p7qLXXJTc3NwwZMgQbN24U909OToaFhQWeeeYZcZtGo8G///1vdOnSRee6dPTo0UZdH9566y0UFBSga9euCAgIwGuvvYajR4+Krzfk2mrOeI+dmXjQN1WJRILNmzdj//792Lp1K37++We8+OKL+Ne//oX9+/fD3t4eHh4eOHLkCH7++Wf89NNP+Omnn7BmzRpERUXVegP/3caPH49PP/0UP/30E0aNGoWNGzeie/fuCAwMFNusW7cOL7zwAkaNGoXXXnsNHh4ekMlkSEhIEG96buj53f1tH6i6kEgkEvz000+QyWQ12tvb29/3fRqqtvcAqr791kXbk/Xqq6+K98jd6+4LfWRkJJYuXYq8vDw4ODjghx9+wIQJE3T+Y3v55ZexZs0azJkzB2FhYXBycoJEIsH48eMb1XP27LPPYsCAAfj222/xyy+/4IMPPsB7772HLVu2YNiwYeIxP/jggzrvjWzunzWZpt27dyM7OxtJSUlISkqq8fr69evx5JNPNut71vd6oXV3z9LdbYcOHYr8/HzMmzcP3bt3h52dHa5du4YXXnihUZ+rqKgozJ49G1evXoVKpcL+/fvFiQbNqSnXpUmTJtW4p1erV69e4t/Hjx+P6OhoHDlyBEFBQdi4cSOGDBki3kcLAO+88w4WLFiAF198EUuWLIGLiwukUinmzJnTqJ/fwIEDcf78eXz//ff45Zdf8MUXX+Df//43Vq1ahalTpzb42mqumNg1A3d3d9ja2uL06dM1Xjt16hSkUil8fX0BAB06dKh1dlJjazp16NABGo0GZ8+e1bnxU6lUoqCgAB06dNBp/+ijj+LRRx/F0qVLsWHDBkycOBFJSUmYOnUqAMDKygojR47EyJEjodFoMHPmTHz66adYsGDBfT8QAwcOhLe3N5KTk9G/f3/s3r1bnBGmtXnzZnTq1AlbtmzRufDeewNubdq0aVNrHbR7v6l27twZgiCgY8eO6Nq16wOPe7eG/B6bQjvMa2lpifDw8Ae2j4yMxOLFi/HNN9/A09MTRUVF4oQUrc2bN2Py5Mn417/+JW4rKytrUu04b29vzJw5EzNnzkROTg769OmDpUuXYtiwYeIN6o6Ojg88h9awUgrVbf369fDw8MDKlStrvLZlyxZ8++23WLVqFWxsbNC5c2ccP378vsfr3LkzDhw4gIqKClhaWtbaRttDde+///vdInGvY8eO4cyZM1i7di2ioqLE7ffeqqL9PD8obqAqGYqNjcX//vc/3L59G5aWljVGNWrToUOHOq9L2tebyt3dHQ4ODlCr1fW6Lo0aNQovvfSSOJpw5swZxMXF6bTZvHkzHn/8cSQmJupsLygo0EkAG0JbhSE6Ohq3bt3CwIEDsWjRIkydOrVB11Zzvi5xKLYZyGQyPPnkk/j+++91yjgolUps2LAB/fv3h6OjIwAgIiICqampOHLkiNguPz+/0TWdhg8fDgBYvny5zvZly5YBAEaMGAGgqgv+3m9r2p4W7fDpjRs3dF6XSqXiN7R7S6fcSyqVYuzYsdi6dSu+/vprVFZW1rhgab9F3h3HgQMHHjjNH6i6mBcWFup0u2dnZ9coH/DMM89AJpNh8eLFNc5XEIQa53hvfPX9PTaFh4cHBg8ejE8//RTZ2dk1Xr+3XEqPHj0QEBCA5ORkJCcnw9vbGwMHDqwR+73nu2LFijp7KO5HrVbXGCbx8PCAj4+P+O8gODgYnTt3xocffohbt27d9xzs7OwA1PxPlszf7du3sWXLFjz99NMYO3ZsjUdMTAyKi4vFUhRjxozBX3/9VWtZEO2/7zFjxiAvL6/Wni5tmw4dOkAmk+G3337Tef3jjz+ud+y1Xa8EQahR/snd3R0DBw7E6tWrkZWVVWs8Wm5ubhg2bBjWrVuH9evX46mnnqpXgjN8+HCkpaXpXCtLSkrw2Wefwc/PD/7+/vU+r7rIZDKMGTMG33zzTa1J6r3XJWdnZ0RERGDjxo1ISkqClZUVRo0aVeOY9/4MNm3a1Oj73O69ftvb2+Ohhx4Sr0sNubaa83WJPXYNsHr16lrvOZs9ezbefvtt7Ny5E/3798fMmTNhYWGBTz/9FCqVCu+//77Y9p///CfWrVuHoUOH4uWXXxbLnbRv3x75+fkN/hYRGBiIyZMn47PPPkNBQQEGDRqEtLQ0rF27FqNGjRJvVl67di0+/vhjjB49Gp07d0ZxcTE+//xzODo6isnh1KlTkZ+fjyeeeALt2rXD5cuXsWLFCgQFBdVrGnhkZCRWrFiB+Ph4BAQE1Njn6aefxpYtWzB69GiMGDECFy9exKpVq+Dv719rcnC38ePHY968eRg9ejReeeUVlJaW4pNPPkHXrl11bsLt3Lkz3n77bcTFxeHSpUsYNWoUHBwccPHiRXz77beYPn26WFevNvX9PTbVypUr0b9/fwQEBGDatGno1KkTlEolUlNTcfXq1Rp1niIjI7Fw4UJYW1tjypQpNYaNnn76aXz99ddwcnKCv78/UlNTsWvXLri6ujY4tuLiYrRr1w5jx45FYGAg7O3tsWvXLhw8eFDsEZRKpfjiiy8wbNgwPPzww4iOjkbbtm1x7do17NmzB46Ojti6dSuAqiQQAN544w2MHz8elpaWGDlypHhhJfP1ww8/oLi4GP/3f/9X6+uPPvqoWKw4MjISr732GjZv3oxx48bhxRdfRHBwMPLz8/HDDz9g1apVCAwMRFRUFL766ivExsYiLS0NAwYMQElJCXbt2oWZM2fib3/7G5ycnDBu3DisWLECEokEnTt3xrZt2xp0f1X37t3RuXNnvPrqq7h27RocHR3xzTff1Hqf2kcffYT+/fujT58+mD59Ojp27IhLly5h+/btOl/igarh2LFjxwIAlixZUq9Y5s+fj//9738YNmwYXnnlFbi4uGDt2rW4ePEivvnmmxrXg8Z69913sWfPHoSGhmLatGnw9/dHfn4+MjIysGvXLuTn5+u0j4yMxKRJk/Dxxx8jIiKiRimpp59+Gm+99Raio6Px2GOP4dixY1i/fr3O5LSG8Pf3x+DBgxEcHAwXFxccOnRILMukVd9ra1BQEGQyGd577z0UFhZCLpfjiSeegIeHR6NiMyotOwnXNGlLW9T1uHLliiAIgpCRkSFEREQI9vb2gq2trfD4448Lf/75Z43jHT58WBgwYIAgl8uFdu3aCQkJCcJHH30kABAUCsV9Y7m3vIcgCEJFRYWwePFioWPHjoKlpaXg6+srxMXF6Uz3zsjIECZMmCC0b99ekMvlgoeHh/D0008Lhw4dEtts3rxZePLJJwUPDw/ByspKaN++vfDSSy8J2dnZ9fo5aTQawdfXVwAgvP3227W+/s477wgdOnQQ5HK50Lt3b2Hbtm21ljLBPeVOBEEQfvnlF6Fnz56ClZWV0K1bN2HdunW1/jwEQRC++eYboX///oKdnZ1gZ2cndO/eXZg1a5Zw+vTpB55HfX6PdZXAqW85EUEQhPPnzwtRUVGCl5eXYGlpKbRt21Z4+umnhc2bN9doe/bsWfHf2759+2q8fvPmTSE6Olpwc3MT7O3thYiICOHUqVNChw4dhMmTJzcoPpVKJbz22mtCYGCg4ODgINjZ2QmBgYHCxx9/XKPt4cOHhWeeeUZwdXUV5HK50KFDB+HZZ58VUlJSdNotWbJEaNu2rSCVSln6pBUZOXKkYG1tLZSUlNTZ5oUXXhAsLS3FEhs3btwQYmJihLZt2wpWVlZCu3bthMmTJ+uU4CgtLRXeeOMN8Zrn5eUljB07VqdMUW5urjBmzBjB1tZWaNOmjfDSSy8Jx48fr7XciZ2dXa2xnTx5UggPDxfs7e0FNzc3Ydq0acJff/1Va2mO48ePC6NHjxacnZ0Fa2troVu3bsKCBQtqHFOlUglt2rQRnJychNu3b9fnxygIQtX1YuzYseLxQ0JChG3btum00X6+N23apLO9vuVEBEEQlEqlMGvWLMHX11f82Q4ZMkT47LPParQtKioSbGxsBADCunXrarxeVlYm/OMf/xC8vb0FGxsboV+/fkJqaqowaNAgYdCgQQ2O7+233xZCQkIEZ2dnwcbGRujevbuwdOlSoby8XKddfa+tn3/+udCpUydBJpOZVekTiSDc525KajFz5szBp59+ilu3btV54ysREZm2yspK+Pj4YOTIkTXuPSNqDrzHzgDuXY7mxo0b+Prrr9G/f38mdUREZuy7775Dbm6uzoQMoubEHjsDCAoKwuDBg9GjRw8olUokJibi+vXrSElJqXFTPBERmb4DBw7g6NGjWLJkCdzc3JqtKDvRvTh5wgCGDx+OzZs347PPPoNEIkGfPn2QmJjIpI6IyEx98sknWLduHYKCgvDll18aOhwyY+yxIyIiIjITvMeOiFq9lStXws/PD9bW1ggNDUVaWtp92y9fvhzdunWDjY0NfH19MXfuXJSVlYmvL1q0CBKJROfRvXt3nWOUlZVh1qxZcHV1hb29PcaMGQOlUqmX8yOi1oOJHRG1asnJyYiNjUV8fDwyMjIQGBiIiIiIOmuebdiwAfPnz0d8fDwyMzORmJiI5ORkvP766zrtHn74YWRnZ4uPffv26bw+d+5cbN26FZs2bcKvv/6K69ev66yzSUTUGGY3FKvRaHD9+nU4ODiY9ZIhRFRV2b+4uBg+Pj6NLtIaGhqKRx55RFzJQKPRwNfXFy+//DLmz59fo31MTAwyMzN1Fq//xz/+gQMHDojJ26JFi/Ddd9/VKE6rVVhYCHd3d2zYsEEsVnvq1Cn06NEDqampePTRR+sVO693RK1DQ651Zjd54vr1682ynicRmY4rV66gXbt2Dd6vvLwc6enpOmtcSqVShIeH17nU3WOPPYZ169YhLS0NISEhuHDhAn788Uc8//zzOu3Onj0LHx8fWFtbIywsDAkJCWjfvj0AID09HRUVFTrrWXbv3h3t27e/b2KnUql0lve7du1asywnRUSmoT7XOrNL7BwcHABUnXxzrOtJRMarqKgIvr6+4ue+ofLy8qBWq+Hp6amz3dPTU1xg/V7PPfcc8vLy0L9/fwiCgMrKSvz973/XGYoNDQ3Fl19+iW7duiE7OxuLFy/GgAEDcPz4cTg4OEChUMDKyqrGEkyenp5QKBR1xpuQkIDFixfX2M7rHZF5a8i1zuwSO+1whKOjIy90RK1ESw5D7t27F++88w4+/vhjhIaG4ty5c5g9ezaWLFmCBQsWAACGDRsmtu/VqxdCQ0PRoUMHbNy4EVOmTGn0e8fFxSE2NlZ8rr3Y83pH1DrU51pndokdEVF9ubm5QSaT1ZiNqlQq4eXlVes+CxYswPPPP4+pU6cCAAICAlBSUoLp06fjjTfeqPX+F2dnZ3Tt2hXnzp0DAHh5eaG8vBwFBQU6vXb3e18AkMvlkMvlDT1NImpFOCuWiFotKysrBAcH60yE0Gg0SElJQVhYWK37lJaW1kjetEsB1jUX7datWzh//jy8vb0BAMHBwbC0tNR539OnTyMrK6vO9yUiqg/22BFRqxYbG4vJkyejb9++CAkJwfLly1FSUoLo6GgAQFRUFNq2bYuEhAQAwMiRI7Fs2TL07t1bHIpdsGABRo4cKSZ4r776KkaOHIkOHTrg+vXriI+Ph0wmw4QJEwAATk5OmDJlCmJjY+Hi4gJHR0e8/PLLCAsLq/eMWCKi2jCxI6JWLTIyErm5uVi4cCEUCgWCgoKwY8cOcUJFVlaWTg/dm2++CYlEgjfffBPXrl2Du7s7Ro4ciaVLl4ptrl69igkTJuDGjRtwd3dH//79sX//fri7u4tt/v3vf0MqlWLMmDFQqVSIiIjAxx9/3HInTkRmqUXq2K1cuRIffPABFAoFAgMDsWLFCoSEhNTadvDgwfj1119rbB8+fDi2b9/+wPcqKiqCk5MTCgsLeTMxkZlr7Z/31n7+RK1FQz7rer/HrqFV3bds2aJTrf348eOQyWQYN26cvkMlIiIiMml6T+yWLVuGadOmITo6Gv7+/li1ahVsbW2xevXqWtu7uLjAy8tLfOzcuRO2trZM7IiIiIgeQK+Jnbaq+93V1R9U1f1eiYmJGD9+POzs7PQVJhEREZFZ0OvkicZUdb9bWloajh8/jsTExDrb3LvETlFRUeMDJiIiIjJhRl3HLjExEQEBAXVOtACqlthxcnISH1wnloiIiForvSZ2janqrlVSUoKkpKQHLr8TFxeHwsJC8XHlypUmx01ERERkivSa2DWmqrvWpk2boFKpMGnSpPu2k8vl4jqJXC+RiIiIWjO9FyhuaFV3rcTERIwaNQqurq76DpGIiIjILOg9sWtoVXegas3Effv24ZdfftF3eERERERmo0VWnmhJDanOnFNchkOXbsJKJkW4v+d92xKR8WntKy+09vMnqotGI6BCo0GFWkClWoNytQaVagEVak314/5/t5BK4OloDW8nG7g7yCGTSgx6Pg35rLfqtWL/ulKImesz0KudExM7IiIiI5RbrMKRKwU4nHUTf10twM2SClRq7p+cqTXN12clk0rg4SCHl5M1vJ2s4eVoU/Vn9XNPx6qHlYVxFBpp1Ymdt5M1ACC7sMzAkRAREVF5pQaZ2UXIyLqJw1kFOHzlJq7k327ycSUSwFImhaVUAksLKSykUljJtH+XVL0mk8JSVvX3crUGysIyKItVUGsEZBeWIbuwDIfv8x5u9nKdhK+2RNDaUtbkc3kQJnYA8m6pUF6pMZpsm4iIqDXILryNw1kFyLh8E4evFODYtUKUV2p02kgkQFcPB/Ru74wgX2d4O9vAUiaBlUwKi7uSMct7/m6hbSOVQCaVQCJp+HCqWiMg75YK2YVlUBTervqzqAyK6kRPUf0oV2uQd0uFvFsqHLtWWOfxnG0t4eWoTfxsENrRBaN6t21wXPfTqhM7FzsrWFlIUV6pgbKoDL4utoYOiYiIyCyVVahx/Frhnd64rAIoimqOmDnbWqK3rzP6tG+D3u3boJevExytLQ0QcdUwrHaoFb7OtbYRBAH5JeViopddVJUEKgpVUBRVJYPZBWW4XaFGQWkFCkorcEpRDKCqh5KJXTOSSCTwdrLG5RulyC5kYkdERNQcBEHAlfzb1UlcVW/cyetFqLzn3jeZVILuXlW9cdpEzs/VtlG9a4YikUjgai+Hq70cPds61dpGEAQUlVVW9/TdFnv8eng3/6SnVp3YAYCXozaxa/oYPhERkTERBAEZWQU4l1MMiUQCCQCpRAKptOpPaJ9LJJBIAKkENdrpPK9uV9VWd7/ScjWOXi2sSuSyCnCjpLxGPG72cvRp74ze7dugT3tnBLRzgq2V+aciEokETjaWcLKxRDcvB72+l/n/NB9Ae5+dghMoiIjITJSWV+L7I9ex9s9L4rBfS7OUSfCwjxN635XItXW2ManeOFPExM7ZBgBnxhIRkem7mFeCr1MvY1P6FRSXVQIArC2lCOnoCpkEEABohKqePI0gQBAAjSBAIwAQ/171XMC97aqeC3e1EwTtMQXIJBL08HFEb9+qRO5hH8cWmQVKupjYiSVPOBRLRESmR60RsPd0DtamXsZvZ3LF7e1dbBEV1gHjgn3hZGuYyQfU8pjYObHHjoiITM/NknIkH7qCdfsv4+rNqs4JiQQY3NUdUY/5YVAXd0gNvGICtTwmdixSTEREJuTo1QJ8lXoZW/+6DlV1zTcnG0tEPuKLiaHt0cHVzsARkiExsWORYiIiMnKqSjW2H83GV6mXceRKgbj9YR9HTA7zw8hAH9hY8X42YmJXVaRYu3wIixQTEZERuVZwG+v3X0bywSti+RBLmQQjArzxfJgf+rR35ixT0tHqEzuJRAIvJ2tk5ZdCwcSOiIgMTBAE/HHuBr5KvYRdmUpoa/p6O1ljYmh7RD7SHu4OcsMGSUar1Sd2QNWHJSu/FNcLODOWiIgMo7isAt+kX8XX+y/jfG6JuP2xzq6ICuuA8B6esJDxdiG6PyZ2YJFiIiIynDPKYnyVeglbMq6htFwNALCzkmFMcDs8/2gHdPHU70oFZF6Y2IFFiomIqGVVqDXYeVKJr1IvYf+FfHH7Qx72iArrgNG928LBQAvfk2ljYgcWKSYiopaRU1yGpLQr2HAgC4qiqs4EqQR40t8LUWEdENbZlZMhqEmY2IFFiomISH8EQUD65Zv4KvUyfjqejQp11WwIVzsrTAhpj+dC28OneuSIqKmY2IFFiomIqPndLlfj+yPX8FXqZZzMLhK392nvjKgwPwwL8ILcgrXnqHkxsQPgxSLFRETUTC7llWDd/svYeOgKisoqAQByCyn+FuSDqDA/9GzrZOAIyZwxsUNVdziLFBMRUWNpNAL2nsnBV6mXsfd0rri9vYstJj3aHuOCfdHGzsqAEVJrwcQOLFJMRESNU1Bajo2HrmDd/ixk5ZeK2wd3c8fkMD8M6uoOqZSTIajlMLGrxiLFRERUX8evFeKr1Ev4/sh1qCo1AABHaws829cXkx7tAD83OwNHSK0VE7tqLFJMRET3o6pU46djCnyVegkZWQXidn9vR0SFdcDfgtrCxoqTIciwmNhVY5FiIiKqzfWC29hwIAtJB7OQd6scAGApk2BYT29EhXVAcIc2rD1HRoOJXTUWKSYiIi1BEJB64Qa++vMydmYqodZU1Z7zcrTGxND2iAzxhYeDtYGjJKqJiV01L0cOxRIREaDWCJiTfARb/7oubnu0kwsmh/kh3N8TljKWxCLjxcSumrbq93UmdkRErZYgCHjzu+PY+td1WMokiHzEF1Fhfujq6WDo0IjqhYldNRYpJiKif/1yBv9Ly4JEAvxnfG8MD/A2dEhEDcLspZq2SLEgAMoi9toREbU2ifsu4r97zgEAlo4KYFJHJomJXTVtkWIAUDCxIyJqVbZkXMWSbScBAK9FdMNzoe0NHBFR4zCxu4uXODOWiR0RUWuRkqnEa5uPAgCm9O+ImYM7GzgiosZjYncXH21ix9UniIhahbSL+Zi5PgNqjYBnerfFG8N7sCYdmTQmdnfxcmKRYqLWaOXKlfDz84O1tTVCQ0ORlpZ23/bLly9Ht27dYGNjA19fX8ydOxdlZXeuGwkJCXjkkUfg4OAADw8PjBo1CqdPn9Y5xuDBgyGRSHQef//73/VyflS7k9eLMGXtQagqNRjS3QPvje3FdV3J5Ok9sWvoBbOgoACzZs2Ct7c35HI5unbtih9//FHfYQIAfJxZpJiotUlOTkZsbCzi4+ORkZGBwMBAREREICcnp9b2GzZswPz58xEfH4/MzEwkJiYiOTkZr7/+utjm119/xaxZs7B//37s3LkTFRUVePLJJ1FSUqJzrGnTpiE7O1t8vP/++3o9V7rj8o0STF6ThuKySjzi1wYrJ/ZhfToyC3otd6K9YK5atQqhoaFYvnw5IiIicPr0aXh4eNRoX15ejqFDh8LDwwObN29G27ZtcfnyZTg7O+szTBGLFBO1PsuWLcO0adMQHR0NAFi1ahW2b9+O1atXY/78+TXa//nnn+jXrx+ee+45AICfnx8mTJiAAwcOiG127Nihs8+XX34JDw8PpKenY+DAgeJ2W1tbeHl56eO06D5yisrwfGIacotV6O7lgC8mPwJrS67xSuZBr19P7r5g+vv7Y9WqVbC1tcXq1atrbb969Wrk5+fju+++Q79+/eDn54dBgwYhMDBQn2GKWKSYqHUpLy9Heno6wsPDxW1SqRTh4eFITU2tdZ/HHnsM6enp4ujDhQsX8OOPP2L48OF1vk9hYSEAwMXFRWf7+vXr4ebmhp49eyIuLg6lpaVNPSV6gMLbFYhanYas/FK0d7HFV1NC4GRjaeiwiJqN3nrstBfMuLg4cduDLpg//PADwsLCMGvWLHz//fdwd3fHc889h3nz5kEm0/+3KRYpJmpd8vLyoFar4enpqbPd09MTp06dqnWf5557Dnl5eejfvz8EQUBlZSX+/ve/6wzF3k2j0WDOnDno168fevbsqXOcDh06wMfHB0ePHsW8efNw+vRpbNmypc54VSoVVCqV+LyoqKghp9vq3S5XY+ragzilKIabvRxfTwnheq9kdvSW2DXmgnnhwgXs3r0bEydOxI8//ohz585h5syZqKioQHx8fK37NOeFzsW2qkhxuVqDnOIytGtj2+hjEZF52rt3L9555x18/PHHCA0Nxblz5zB79mwsWbIECxYsqNF+1qxZOH78OPbt26ezffr06eLfAwIC4O3tjSFDhuD8+fPo3Ln2chsJCQlYvHhx855QK1Gh1iBmQwYOXroJB2sLfPViCDq42hk6LKJmZ1RdUhqNBh4eHvjss88QHByMyMhIvPHGG1i1alWd+yQkJMDJyUl8+Pr6Nvr9pVIJa9kRtSJubm6QyWRQKpU625VKZZ33vi1YsADPP/88pk6dioCAAIwePRrvvPMOEhISoNFodNrGxMRg27Zt2LNnD9q1a3ffWEJDQwEA586dq7NNXFwcCgsLxceVK1fqc5qtnkYjYN7mo0g5lQO5hRSJkx+Bv4+jocMi0gu9JXaNuWB6e3uja9euOsOuPXr0gEKhQHl5ea37NPeFjokdUethZWWF4OBgpKSkiNs0Gg1SUlIQFhZW6z6lpaWQSnUvndprliAI4p8xMTH49ttvsXv3bnTs2PGBsRw5cgRA1XWwLnK5HI6OjjoPuj9BEPD29kxsOXwNMqkEH0/sg5COLg/ekchE6S2xa8wFs1+/fjh37pzOt94zZ87A29sbVlZWte7T3Bc6Fikmal1iY2Px+eefY+3atcjMzMSMGTNQUlIizpKNiorSuVd45MiR+OSTT5CUlISLFy9i586dWLBgAUaOHCkmeLNmzcK6deuwYcMGODg4QKFQQKFQ4PbtquvK+fPnsWTJEqSnp+PSpUv44YcfEBUVhYEDB6JXr14t/0MwYx/vPY/Vf1wEAHwwtheG9PB8wB5Epk2v5U5iY2MxefJk9O3bFyEhIVi+fHmNC2bbtm2RkJAAAJgxYwb++9//Yvbs2Xj55Zdx9uxZvPPOO3jllVf0GaYOFikmal0iIyORm5uLhQsXQqFQICgoCDt27BDvD87KytLpoXvzzTchkUjw5ptv4tq1a3B3d8fIkSOxdOlSsc0nn3wCoKoI8d3WrFmDF154AVZWVti1a5d4TfT19cWYMWPw5ptv6v+EW5ENB7Lwwc9VhaEXPO2PZ/rcfzicyBzoNbFr6AXT19cXP//8M+bOnYtevXqhbdu2mD17NubNm6fPMHVoixSzlh1R6xETE4OYmJhaX9u7d6/OcwsLC8THx9c5oQu4MyRbF19fX/z6668NjpPq78dj2Xjju2MAgFmPd8aU/g8eDicyB3pN7ICGXTABICwsDPv379dzVHXTFinm6hNERKZp39k8zEk6AkEAJoS0x6tPdjN0SEQtxqhmxRoDbw7FEhGZrL+uFGD614dQrtZgeIAX3h7VExIJ13+l1oOJ3T28q4dic6uLFBMRkWk4l3MLL6xJQ2m5Gv0ecsW/I4MgkzKpo9aFid09tEWKBQHIKWavHRGRKbhecBvPJx7AzdIKBLZzwqfP94Xcguu/UuvDxO4eLFJMRGRa8kvK8XziAWQXlqGTux3WRIfAXq73W8iJjBITu1owsSMiMg23VJWIXpOG87kl8HayxtdTQuFiV3vdU6LWgIldLbydtCVPODOWiMhYqSrV+PvX6fjraiHa2Fri6ykhaOtsY+iwiAyKiV0ttDNjrxewx46IyBipNQLmJh/BvnN5sLWSYU10CB7ycDB0WEQGx8SuFnd67JjYEREZG0EQsOD74/jxmAKWMgk+e74vgnydDR0WkVFgYlcLbycWKSYiMlb/3nkGGw5kQSIBlkf2Rv8uboYOichoMLGrBYsUExEZp3M5t/DR7nMAgLdH9cSIXt4GjojIuDCxqwWLFBMRGadN6VcAAE9098DE0A4GjobI+DCxqwWLFBMRGZ8KtQbfpF8DAEQ+4mvgaIiMExO7WkilEng6yQFwAgURkbHYezoXebdUcLO3whPdPQwdDpFRYmJXB7HkCRM7IiKjsPFQ1TDs6N5tYSnjf19EteEnow4sUkxEZDxyisuw+1QOAGBcXw7DEtWFiV0dWKSYiMh4fHf4GtQaAUG+zujqyULERHVhYlcHFikmIjIOgiBg46GrADhpguhBmNjVQSxSXMTEjojIkDKyCnAu5xasLaV4mnXriO6LiV0dxCLFBbzHjojIkDZVT5oYHuANB2tLA0dDZNyY2NXBy+lOkeIKNYsUExEZQml5Jbb+dR0A8CwnTRA9EBO7Orja3SlSrORwLBGRQfx4TIGScjU6uNoitKOLocMhMnpM7OrAIsVERIa38WDVMOyzfX0hkUgMHA2R8WNidx8sUkxEZDgXcm8h7VI+pBLgmT5tDR0OkUlgYncfLFJMRGQ4m9OrSpwM7OouftEmovtjYncf4sxY9tgREbWoSrUG32RUJXacNEFUf0zs7kOsZcfVJ4iIWtTvZ/OgLFLBxc4K4T08DR0OkclgYncfXixSTERkEMnVkyZGBbWFlQX/qyKqL35a7sOHRYqJiFrcjVsq7MpUAgCefaSdgaMhMi1M7O6DRYqJiFret4evoVIjoFc7J3T3cjR0OEQmhYndfbBIMRFRyxIEARurlxAbx0kTRA3GxO4+WKSYiKhlHb1aiDPKW5BbSPF/gT6GDofI5DCxewBvR5Y8ISJqKcnVvXXDenrBycbSwNEQmR4mdg/g7Vw9M5ZFiomI9Op2uRpbj1wHwNp1RI3FxO4BxJIn7LEjItKrHSeyUayqRLs2Nni0k6uhwyEySUzsHuBOyRMmdkRE+rTxYNVKE+OCfSGVSgwcDZFpYmL3ACxSTESkf5dvlCD1wg1IJMDYvqxdR9RYLZLYrVy5En5+frC2tkZoaCjS0tLqbPvll19CIpHoPKytrVsizFppe+wUvMeOiEhvNqdX9db1f8gNbZ1tDBwNkenSe2KXnJyM2NhYxMfHIyMjA4GBgYiIiEBOTk6d+zg6OiI7O1t8XL58Wd9h1knbY5dTzCLFRET6oNYIYmLHSRNETaP3xG7ZsmWYNm0aoqOj4e/vj1WrVsHW1harV6+ucx+JRAIvLy/x4elpuAWgXe2sYCmTQBCqkjsiImpe+87lIbuwDE42lhjqb7jrPZE50GtiV15ejvT0dISHh995Q6kU4eHhSE1NrXO/W7duoUOHDvD19cXf/vY3nDhxos62KpUKRUVFOo/mJJVK7txnxzVjiYianXalidG928LaUmbgaIhMm14Tu7y8PKjV6ho9bp6enlAoFLXu061bN6xevRrff/891q1bB41Gg8ceewxXr16ttX1CQgKcnJzEh69v83fjs0gxEZF+3Cwpx84TSgDAOE6aIGoyo5sVGxYWhqioKAQFBWHQoEHYsmUL3N3d8emnn9baPi4uDoWFheLjypUrzR4TixQTEenHd0euoVytwcM+jnjYx8nQ4RCZPAt9HtzNzQ0ymQxKpVJnu1KphJeXV72OYWlpid69e+PcuXO1vi6XyyGXy5sc6/2wSDERUfMTBAHJB6u+jHPSBFHz0GuPnZWVFYKDg5GSkiJu02g0SElJQVhYWL2OoVarcezYMXh7e+srzAe6U/KEiR2ROWpISSYAWL58Obp16wYbGxv4+vpi7ty5KCvTvT486JhlZWWYNWsWXF1dYW9vjzFjxtT4EmzuTlwvwilFMawspPhbkI+hwyEyC3ofio2NjcXnn3+OtWvXIjMzEzNmzEBJSQmio6MBAFFRUYiLixPbv/XWW/jll19w4cIFZGRkYNKkSbh8+TKmTp2q71DrpO2xu87EjsjsNLQk04YNGzB//nzEx8cjMzMTiYmJSE5Oxuuvv96gY86dOxdbt27Fpk2b8Ouvv+L69et45pln9H6+xkTbWxfxsBecba0MHA2RmRBawIoVK4T27dsLVlZWQkhIiLB//37xtUGDBgmTJ08Wn8+ZM0ds6+npKQwfPlzIyMio93sVFhYKAITCwsJmi/+vKzeFDvO2CSFLdzbbMYmo6Zrj8x4SEiLMmjVLfK5WqwUfHx8hISGh1vazZs0SnnjiCZ1tsbGxQr9+/ep9zIKCAsHS0lLYtGmT2CYzM1MAIKSmptY7dn1c71rK7fJKISB+h9Bh3jbhtzM5hg6HyKg15LPeIpMnYmJicPnyZahUKhw4cAChoaHia3v37sWXX34pPv/3v/8ttlUoFNi+fTt69+7dEmHWybt6KJZFionMS2NKMj322GNIT08Xh1YvXLiAH3/8EcOHD6/3MdPT01FRUaHTpnv37mjfvv19S0GZk59PKFBUVom2zjZ4rLObocMhMht6nTxhLrRFiivUAnKKVVzuhshM3K8k06lTp2rd57nnnkNeXh769+8PQRBQWVmJv//97+JQbH2OqVAoYGVlBWdn5xpt6ioFBVTV7VSp7hRKb+66nS1p06GqElZjgttBJpUYOBoi82F05U6MEYsUE5HW3r178c477+Djjz9GRkYGtmzZgu3bt2PJkiV6f++WqNvZEq7kl+KP83kAgHHBrF1H1JyY2NUTixQTmZ/GlGRasGABnn/+eUydOhUBAQEYPXo03nnnHSQkJECj0dTrmF5eXigvL0dBQUG93xdombqdLWFz+lUIAtDvIVf4utgaOhwis8LErp60RYpZ8oTIfDSmJFNpaSmkUt1Lp0xWtQyWIAj1OmZwcDAsLS112pw+fRpZWVn3LQUll8vh6Oio8zA1Go2AzelVw7CsXUfU/HiPXT3dKXnCoVgicxIbG4vJkyejb9++CAkJwfLly2uUZGrbti0SEhIAACNHjsSyZcvQu3dvhIaG4ty5c1iwYAFGjhwpJngPOqaTkxOmTJmC2NhYuLi4wNHRES+//DLCwsLw6KOPGuYH0UL+PH8D1wpuw8HaAhEP169QPRHVHxO7evJ2ZI8dkTmKjIxEbm4uFi5cCIVCgaCgIOzYsUOc/JCVlaXTQ/fmm29CIpHgzTffxLVr1+Du7o6RI0di6dKl9T4mUFUBQCqVYsyYMVCpVIiIiMDHH3/cciduIBsPVQ0f/y3IB9aWMgNHQ2R+JIIgCIYOojkVFRXByckJhYWFzTpM8fMJBV76Oh2Bvs74fla/ZjsuETWevj7vpsLUzr+wtAKPvLML5ZUabI3pj4B2XBuWqD4a8lnnPXb15O2k7bHjUCwRUWN8/9c1lFdq0N3LAT3bGn8iSmSKmNjVE4sUExE1jXYY9tm+vpBIWLuOSB+Y2NWTtkixIFQld0REVH8nrhfi+LUiWMokGNW7raHDITJbTOzqSSqVwNORw7FERI2hXWliqL8nXOysDBwNkfliYtcAPtXDsdcLODOWiKi+VJVqfHfkGgDWriPSNyZ2DeDlxJInREQNtfOkEgWlFfBytMaALu6GDofIrDGxawDt6hMsUkxEVH8bq4dhxwa3g0zKSRNE+sTErgFYpJiIqGGuF9zG72dzAQDj+rYzcDRE5o+JXQN4O1fdY5fNxI6IqF6+Sb8KQQAe7eSCDq52hg6HyOwxsWsAbZHibA7FEhE9kEYjYGP6ndp1RKR/TOwaQDt5gkWKiYgebP/FG7iSfxv2cgsM6+lt6HCIWgUmdg3gZidnkWIionrS1q4bGegDGyuZgaMhah2Y2DUAixQTEdVPUVkFfjyWDQB4lpMmiFoME7sGYpFiIqIH2/rXdagqNejqaY8gX2dDh0PUajCxayAWKSYierCNB+9MmpBIWLuOqKUwsWsgbZFiljwhIqrdKUUR/rpaCAupBKN6tzV0OEStChO7BtIWKWbJEyKi2mknTQzp4QE3e7mBoyFqXZjYNZCXE4sUExHVpbxSg28PXwPA2nVEhsDEroF8nNljR0RUl5RMJfJLyuHhIMegru6GDoeo1WFi10AsUkxEVLeNh6omTYwJbgcLGf+LIWpp/NQ1EIsUExHVTlFYhl/P5AIAxgWzdh2RITCxayAWKSYiqt2O49nQCEDfDm3Qyd3e0OEQtUpM7BrBhxMoiIhquHKz6stucIc2Bo6EqPViYtcI2vvssrn6BBGRSFFUdU30qB7VIKKWx8SuEbydWKSYiOheOdWJnacja9cRGQoTu0a4k9jxHjsiIi1lUdWEMi/22BEZDBO7RmCRYiIiXYIgiEOxnkzsiAyGiV0jaIsUK5jYEREBAApvV6C8sqq2pweHYokMpkUSu5UrV8LPzw/W1tYIDQ1FWlpavfZLSkqCRCLBqFGj9BtgA90pUlyGShYpJiISe+va2FpCbiEzcDRErZfeE7vk5GTExsYiPj4eGRkZCAwMREREBHJycu6736VLl/Dqq69iwIAB+g6xwbRFijUsUkxEBODO/XUchiUyLL0ndsuWLcO0adMQHR0Nf39/rFq1Cra2tli9enWd+6jVakycOBGLFy9Gp06d9B1ig91dpJgTKIiIACXvryMyCnpN7MrLy5Geno7w8PA7byiVIjw8HKmpqXXu99Zbb8HDwwNTpkzRZ3hNwpInRER3KAtZ6oTIGFjo8+B5eXlQq9Xw9PTU2e7p6YlTp07Vus++ffuQmJiII0eO1Os9VCoVVKo7w6FFRUWNjrchvJ1sANxkkWIiIgDKYvbYERkDo5oVW1xcjOeffx6ff/453Nzc6rVPQkICnJycxIevr6+eo6zCHjsiojt4jx2RcdBrj52bmxtkMhmUSqXOdqVSCS8vrxrtz58/j0uXLmHkyJHiNo2mataphYUFTp8+jc6dO+vsExcXh9jYWPF5UVFRiyR32sROUcR77IiIeI8dkXHQa2JnZWWF4OBgpKSkiCVLNBoNUlJSEBMTU6N99+7dcezYMZ1tb775JoqLi/Gf//yn1oRNLpdDLm/5ezq0RYqvcyiWiEhM7LjqBJFh6TWxA4DY2FhMnjwZffv2RUhICJYvX46SkhJER0cDAKKiotC2bVskJCTA2toaPXv21Nnf2dkZAGpsNzSxx45DsUTUylWqNcgt1g7FcvIEkSHpPbGLjIxEbm4uFi5cCIVCgaCgIOzYsUOcUJGVlQWp1Khu9asXb2fdIsUWMtM7ByKi5nCjpBwaAZBJJXC1Z2JHZEh6T+wAICYmptahVwDYu3fvfff98ssvmz+gZqAtUlyhFpBTrIKPs42hQyIiMgjtMKy7vRwyqcTA0RC1buxmaiQWKSYiqqJgDTsio8HErglY8oSICFAWs9QJkbFgYtcE3tUzYzmBgohaszurTjCxIzI0JnZNoO2xY8kTImrN7tSw41AskaExsWsCLxYpJiLiUCyREWFi1wTeLFJMRMShWCIjwsSuCVikmMh8rFy5En5+frC2tkZoaCjS0tLqbDt48GBIJJIajxEjRohtantdIpHggw8+ENv4+fnVeP3dd9/V63nqg7KYiR2RsWiROnbmikWKicxDcnIyYmNjsWrVKoSGhmL58uWIiIjA6dOn4eHhUaP9li1bUF5eLj6/ceMGAgMDMW7cOHFbdna2zj4//fQTpkyZgjFjxuhsf+uttzBt2jTxuYODQ3OdVosoq1CjoLQCAJcTIzIGTOyawM1ODgupBJUaFikmMmXLli3DtGnTxKUOV61ahe3bt2P16tWYP39+jfYuLi46z5OSkmBra6uT2Hl5eem0+f777/H444+jU6dOOtsdHBxqtDUlOUVV99fJLaRwtOF/KUSGxi6mJtAtUszhWCJTVF5ejvT0dISHh4vbpFIpwsPDkZqaWq9jJCYmYvz48bCzs6v1daVSie3bt2PKlCk1Xnv33Xfh6uqK3r1744MPPkBlZWWd76NSqVBUVKTzMDTtMKyXkzUkEq46QWRo/HrVRD7O1rhWcLt69Yk2hg6HiBooLy8ParVaXL9ay9PTE6dOnXrg/mlpaTh+/DgSExPrbLN27Vo4ODjgmWee0dn+yiuvoE+fPnBxccGff/6JuLg4ZGdnY9myZbUeJyEhAYsXL67HWbUccdUJBw7DEhkDJnZN5OVkA+AmJ1AQtVKJiYkICAhASEhInW1Wr16NiRMnwtpaN/mJjY0V/96rVy9YWVnhpZdeQkJCAuTymjXh4uLidPYpKiqCr69vM5xF42lr2Hmwhh2RUeBQbBP5sEgxkUlzc3ODTCaDUqnU2a5UKh9471tJSQmSkpJqHWLV+v3333H69GlMnTr1gbGEhoaisrISly5dqvV1uVwOR0dHnYeh5VTXsOPECSLjwMSuiVikmMi0WVlZITg4GCkpKeI2jUaDlJQUhIWF3XffTZs2QaVSYdKkSXW2SUxMRHBwMAIDAx8Yy5EjRyCVSmudiWusFKxhR2RUOBTbRNoixZw8QWS6YmNjMXnyZPTt2xchISFYvnw5SkpKxFmyUVFRaNu2LRISEnT2S0xMxKhRo+Dq6lrrcYuKirBp0yb861//qvFaamoqDhw4gMcffxwODg5ITU3F3LlzMWnSJLRpYzr363Iolsi4MLFrIm2R4mwOxRKZrMjISOTm5mLhwoVQKBQICgrCjh07xAkVWVlZkEp1BzhOnz6Nffv24ZdffqnzuElJSRAEARMmTKjxmlwuR1JSEhYtWgSVSoWOHTti7ty5OvfQmQIOxRIZF4kgCIKhg2hORUVFcHJyQmFhYYvcf5JTVIaQd1IglQBn3h7GIsVELailP+/GxtDnLwgC/Bf+jNsVaux9dTD83Gov90JETdOQzzqzkCZys68qUqwR7nxzJSJqDYpVlbhdoQbAe+yIjAUTuyZikWIiaq2U1dc8R2sL2FjJDBwNEQFM7JqFj7M2sePMWCJqPZTVy4mxt47IeDCxawZe1TNjWaSYiFoT7YxYbdknIjI8JnbNQFukmEOxRNSaKLSlTricGJHRYGLXDLycOBRLRK1Pjthjxxp2RMaCiV0z8GaPHRG1QrzHjsj4MLFrBuLqEyxSTEStCIdiiYwPE7tmoO2xyykuQ6VaY+BoiIhaRg4nTxAZHSZ2zeDuIsW5t1ikmIjMn0YjiEXZPblOLJHRYGLXDO4uUnydw7FE1ArcKClHpUaARFL15ZaIjAMTu2aiLVLMWnZE1Bpoa9i52cthyTWyiYwGP43NRFukmCVPiKg10CZ2HIYlMi5M7JoJS54QUWuiLXXixVInREaFiV0z8WaRYiJqRcRSJ0zsiIwKE7tmwh47ImpNtKVOPFnDjsioMLFrJtoixZw8QUStgZLLiREZJSZ2zUTbY6csYpFiIjJ/iup77DgUS2RcmNg1E1cWKSaiVkRcdYKJHZFRaZHEbuXKlfDz84O1tTVCQ0ORlpZWZ9stW7agb9++cHZ2hp2dHYKCgvD111+3RJhNImORYiJqJcorNbhRUg4A4nWPiIyD3hO75ORkxMbGIj4+HhkZGQgMDERERARycnJqbe/i4oI33ngDqampOHr0KKKjoxEdHY2ff/5Z36E2mXY4lvfZEZE5yymuusZZyaRoY2tp4GiI6G56T+yWLVuGadOmITo6Gv7+/li1ahVsbW2xevXqWtsPHjwYo0ePRo8ePdC5c2fMnj0bvXr1wr59+/QdapN5O7NIMRGZP6V4f50cEonEwNEQ0d30mtiVl5cjPT0d4eHhd95QKkV4eDhSU1MfuL8gCEhJScHp06cxcODAWtuoVCoUFRXpPAyFJU+IqDW4s+oEh2GJjI1eE7u8vDyo1Wp4enrqbPf09IRCoahzv8LCQtjb28PKygojRozAihUrMHTo0FrbJiQkwMnJSXz4+vo26zk0BIdiiag14HJiRMbLKGfFOjg44MiRIzh48CCWLl2K2NhY7N27t9a2cXFxKCwsFB9Xrlxp2WDvok3srnMolojMmHYolj12RMbHQp8Hd3Nzg0wmg1Kp1NmuVCrh5eVV535SqRQPPfQQACAoKAiZmZlISEjA4MGDa7SVy+WQy43jW6MXixQTUSvAoVgi46XXHjsrKysEBwcjJSVF3KbRaJCSkoKwsLB6H0ej0UClMv7acD4sUkxErYCSNeyIjJZee+wAIDY2FpMnT0bfvn0REhKC5cuXo6SkBNHR0QCAqKgotG3bFgkJCQCq7pnr27cvOnfuDJVKhR9//BFff/01PvnkE32H2mTaIsWVGgG5t1TiMmNEROZEm9h58B47IqOj98QuMjISubm5WLhwIRQKBYKCgrBjxw5xQkVWVhak0jsdhyUlJZg5cyauXr0KGxsbdO/eHevWrUNkZKS+Q20ybZHiawW3cb2gjIkdEZkl3mNHZLz0ntgBQExMDGJiYmp97d5JEW+//TbefvvtFohKP7ydqhI73mdHRObolqoSt1SVAJjYERkjo5wVa8pYpJiIzJl2GNZebgF7eYv0DRBRAzCxa2YsUkxE5oz31xEZNyZ2zUw7S4xDsURkjnKq76/jjFgi48TErpn5OLNIMRGZLwVr2BEZNSZ2zYxFionInLE4MZFxY2LXzLRFinOKVSxSTERmJ0csdcJ77IiMERO7ZqYtUqyuLlJMRGROOBRLZNyY2DUzbZFigDNjicj8cCiWyLgxsdMDseRJARM7IjIfgiBwKJbIyDGx0wMvsZYdZ8YSkfm4WVqB8up7hz0c2GNHZIyY2OmBj7j6BHvsiMh8aIdhXe2sYGXB/z6IjBE/mXrAIsVEZI4U4qoT7K0jMlZM7PRAW6SYQ7FEZE5yqhM7L95fR2S0mNjpgbZIMYdiicicKMWJE+yxIzJWTOz0wJtFiolMzsqVK+Hn5wdra2uEhoYiLS2tzraDBw+GRCKp8RgxYoTY5oUXXqjx+lNPPaVznPz8fEycOBGOjo5wdnbGlClTcOvWLb2dY1NxKJbI+DGx0wM3FikmMinJycmIjY1FfHw8MjIyEBgYiIiICOTk5NTafsuWLcjOzhYfx48fh0wmw7hx43TaPfXUUzrt/ve//+m8PnHiRJw4cQI7d+7Etm3b8Ntvv2H69Ol6O8+mujMUy8SOyFgxsdMDFikmMi3Lli3DtGnTEB0dDX9/f6xatQq2trZYvXp1re1dXFzg5eUlPnbu3AlbW9saiZ1cLtdp16ZNG/G1zMxM7NixA1988QVCQ0PRv39/rFixAklJSbh+/bpez7ex7qw6wXvsiIwVEzs9YZFiItNQXl6O9PR0hIeHi9ukUinCw8ORmppar2MkJiZi/PjxsLOz09m+d+9eeHh4oFu3bpgxYwZu3LghvpaamgpnZ2f07dtX3BYeHg6pVIoDBw408az0g/fYERk/C0MHYK5YpJjINOTl5UGtVsPT01Nnu6enJ06dOvXA/dPS0nD8+HEkJibqbH/qqafwzDPPoGPHjjh//jxef/11DBs2DKmpqZDJZFAoFPDw8NDZx8LCAi4uLlAoFLW+l0qlgkp15/aOoqKi+p5mk1WqNci7xcSOyNgxsdMTbZFi1rIjMm+JiYkICAhASEiIzvbx48eLfw8ICECvXr3QuXNn7N27F0OGDGnUeyUkJGDx4sVNirexcm+pIAiAhVQCVzsrg8RARA/GoVg98eI9dkQmwc3NDTKZDEqlUme7UqmEl5fXffctKSlBUlISpkyZ8sD36dSpE9zc3HDu3DkAgJeXV43JGZWVlcjPz6/zfePi4lBYWCg+rly58sD3bS7aYVgPBzmkUkmLvS8RNQwTOz3x5lAskUmwsrJCcHAwUlJSxG0ajQYpKSkICwu7776bNm2CSqXCpEmTHvg+V69exY0bN+Dt7Q0ACAsLQ0FBAdLT08U2u3fvhkajQWhoaK3HkMvlcHR01Hm0FO3oA0udEBk3JnZ64s31YolMRmxsLD7//HOsXbsWmZmZmDFjBkpKShAdHQ0AiIqKQlxcXI39EhMTMWrUKLi6uupsv3XrFl577TXs378fly5dQkpKCv72t7/hoYceQkREBACgR48eeOqppzBt2jSkpaXhjz/+QExMDMaPHw8fHx/9n3QD5RRzRiyRKeA9dnpyb5FiCxlzaCJjFRkZidzcXCxcuBAKhQJBQUHYsWOHOKEiKysLUqnuZ/j06dPYt28ffvnllxrHk8lkOHr0KNauXYuCggL4+PjgySefxJIlSyCX30mM1q9fj5iYGAwZMgRSqRRjxozBRx99pN+TbSQla9gRmQQmdnqiLVJcqRGQd6tcnCVLRMYpJiYGMTExtb62d+/eGtu6desGQRBqbW9jY4Off/75ge/p4uKCDRs2NChOQ1EUVt9jx8SOyKixG0lP7i5SfJ332RGRidMOxbLHjsi4MbHTI+1wLEueEJGpU4qrTjCxIzJmTOz0SDv8er2APXZEZNq0X1A5eYLIuDGx0yP22BGRObhdrkZRWSUAwJP3CxMZNSZ2euTtxJInRGT6tMOwNpYyOMg5547ImDGx0yMWKSYicyCWOnGyhkTCVSeIjBkTOz3y5nqxRGQGlMV3lhMjIuPGxE6PtD12ymIV1Jra610RERk7ZSFnxBKZCiZ2eqQtUqzWCMit/sZLRGRq7h6KJSLjxsROj1ikmIjMAYdiiUxHiyR2K1euhJ+fH6ytrREaGoq0tLQ6237++ecYMGAA2rRpgzZt2iA8PPy+7Y2dF0ueEJGJ41AskenQe2KXnJyM2NhYxMfHIyMjA4GBgYiIiEBOTk6t7ffu3YsJEyZgz549SE1Nha+vL5588klcu3ZN36HqhTeLFBORiVMWcyiWyFToPbFbtmwZpk2bhujoaPj7+2PVqlWwtbXF6tWra22/fv16zJw5E0FBQejevTu++OILaDQapKSk6DtUvWCRYiIyZYIg3Fl1woGJHZGx02tiV15ejvT0dISHh995Q6kU4eHhSE1NrdcxSktLUVFRARcXl1pfV6lUKCoq0nkYE7FIcRETOyIyPUW3K6Gq1AAAPLicGJHR02til5eXB7VaDU9PT53tnp6eUCgU9TrGvHnz4OPjo5Mc3i0hIQFOTk7iw9fXt8lxNyexSDGHYonIBGmHYZ1tLWFtKTNwNET0IEY9K/bdd99FUlISvv32W1hb1z4EEBcXh8LCQvFx5cqVFo7y/jh5gohMGYdhiUyLXhf9c3Nzg0wmg1Kp1NmuVCrh5eV1330//PBDvPvuu9i1axd69epVZzu5XA653HiHB3yqV5/QFimWSbkcDxGZDm0NO09OnCAyCXrtsbOyskJwcLDOxAftRIiwsLA693v//fexZMkS7NixA3379tVniHrHIsVEZMpyqq9bnqxhR2QS9D4UGxsbi88//xxr165FZmYmZsyYgZKSEkRHRwMAoqKiEBcXJ7Z/7733sGDBAqxevRp+fn5QKBRQKBS4deuWvkPVC5lUgvYutgCAv64WGDYYIqIG0g7FstQJkWnQ61AsAERGRiI3NxcLFy6EQqFAUFAQduzYIU6oyMrKglR6J7/85JNPUF5ejrFjx+ocJz4+HosWLdJ3uHoxqJs7LuSVYNdJJSIevv8QNBGRMdEOxXqwODGRSdB7YgcAMTExiImJqfW1vXv36jy/dOmS/gNqYUN7eGLNH5ew+1QO77MjIpMi3mPHoVgik2DUs2LNxSMdXeBobYEbJeU4cuWmocMhIqo3ZVHVPXYciiUyDUzsWoClTIrHu3sAAHaerH0pNSIiY6PWCMi9VT15gkOxRCaBiV0LCe9RdU/hzpP1K8xMRGRoN25VlWmSSgBXOytDh0NE9cDEroUM6uYOS5kE53NLcCHXNGf4ElHroh2GdXeQw0LG/y6ITAE/qS3E0doSj3ZyBQCkZHI4loiMn0I7cYLDsEQmg4ldCxKHYzOVD2hJRGR4SiZ2RCaHiV0LGtKjagLFoUv5yC8pN3A0RET3lyMmdix1QmQqmNi1oHZtbNHD2xEaAdhzisOxRGTcxKFYB/bYEZkKJnYtbGh1r90uDscSkZHTTp7wZA07IpPBxK6FDfWvWlLs1zO5KKtQGzgaIqK68R47ItPDxK6F9WzrCE9HOUrL1dh/4YahwyEiqpOS99gRmRwmdi1MIpHcVayYw7FEZJxUlWrcLK0AAHixx47IZDCxM4Bw/6rEblemEoIgGDgaIqKacqrvr7OykMLJxtLA0RBRfTGxM4DHOrvCzkoGZZEKx68VGTocIqIatMOwXo7WkEgkBo6GiOqLiZ0ByC1kGNjVHQCLFRORcRJnxPL+OiKTwsTOQHifHREZM20NOw/eX0dkUpjYGcjj3T0glQCZ2UW4erPU0OEQEenIuWsolohMBxM7A3Gxs0JfPxcAQEomV6EgIuPCUidEpomJnQEN7XFndiwRkTFRsDgxkUliYmdA2rIn+y/cQFFZhYGjISK6I0ecPMHEjsiUMLEzoI5udujsbocKtYBfT+caOhwiIgCAIAjssSMyUUzsDOzuYsVERMbglqoSpeVVa1nzHjsi08LEzsCerE7s9pzKQYVaY+BoiIju1LBzsLaArZWFgaMhooZgYmdgQb5t4GpnhaKyShy8lG/ocIiI7poRy2FYIlPDxM7AZFIJnujuAYDFiokMaeXKlfDz84O1tTVCQ0ORlpZWZ9vBgwdDIpHUeIwYMQIAUFFRgXnz5iEgIAB2dnbw8fFBVFQUrl+/rnMcPz+/Gsd499139Xqe9aFkDTsik8XEzgjcfZ+dIAgGjoao9UlOTkZsbCzi4+ORkZGBwMBAREREICen9hqTW7ZsQXZ2tvg4fvw4ZDIZxo0bBwAoLS1FRkYGFixYgIyMDGzZsgWnT5/G//3f/9U41ltvvaVzrJdfflmv51of2qFYD95fR2RyePOEERjQxQ1yCymu5N/GGeUtdPNyMHRIRK3KsmXLMG3aNERHRwMAVq1ahe3bt2P16tWYP39+jfYuLi46z5OSkmBraysmdk5OTti5c6dOm//+978ICQlBVlYW2rdvL253cHCAl5dXc59Sk7DHjsh0scfOCNhaWaD/Q24AODuWqKWVl5cjPT0d4eHh4japVIrw8HCkpqbW6xiJiYkYP3487Ozs6mxTWFgIiUQCZ2dnne3vvvsuXF1d0bt3b3zwwQeorKys8xgqlQpFRUU6D33gPXZEpouJnZHQDsf+wvvsiFpUXl4e1Go1PD09dbZ7enpCoVA8cP+0tDQcP34cU6dOrbNNWVkZ5s2bhwkTJsDR0VHc/sorryApKQl79uzBSy+9hHfeeQf//Oc/6zxOQkICnJycxIevr289zrDhFFxOjMhkcSjWSAypnkDx15UC5BSVwYPflIlMQmJiIgICAhASElLr6xUVFXj22WchCAI++eQTnddiY2PFv/fq1QtWVlZ46aWXkJCQALm8ZlIVFxens09RUZFekjuuOkFkuthjZyQ8HK0R6OsMAEg5VfsN20TU/Nzc3CCTyaBU6vaWK5XKB977VlJSgqSkJEyZMqXW17VJ3eXLl7Fz506d3rrahIaGorKyEpcuXar1dblcDkdHR51Hc9NoBOQUcyiWyFQxsTMi2mLFuzgcS9RirKysEBwcjJSUFHGbRqNBSkoKwsLC7rvvpk2boFKpMGnSpBqvaZO6s2fPYteuXXB1dX1gLEeOHIFUKoWHh0fDT6SZ5JeWo0ItQCIB3B04FEtkajgUa0TCe3jig59PY9+5PJSWV7LiO1ELiY2NxeTJk9G3b1+EhIRg+fLlKCkpEWfJRkVFoW3btkhISNDZLzExEaNGjaqRtFVUVGDs2LHIyMjAtm3boFarxfv1XFxcYGVlhdTUVBw4cACPP/44HBwckJqairlz52LSpElo06ZNy5x4LbQTJ1zt5LCU8bs/kalh5mBEunraw9fFBlfyb+P3s3mIeNi4SiAQmavIyEjk5uZi4cKFUCgUCAoKwo4dO8QJFVlZWZBKdZOc06dPY9++ffjll19qHO/atWv44YcfAABBQUE6r+3ZsweDBw+GXC5HUlISFi1aBJVKhY4dO2Lu3Lk699AZwp3769hbR2SKmNgZEYlEgvAenljzxyXsOqlkYkfUgmJiYhATE1Pra3v37q2xrVu3bnUWFPfz83tgsfE+ffpg//79DY5T3xSsYUdk0vTez96QZXpOnDiBMWPGiMvsLF++XN/hGZ2h1ffZ7T6VA7WGq1AQUcvSDsVyZj6RadJrYtfQZXpKS0vRqVMnvPvuu0ZXib2lPOLnAkdrC9woKceRKzcNHQ4RtTJK1rAjMml6TezuXqbH398fq1atgq2tLVavXl1r+0ceeQQffPABxo8fX2sNp9bAUibF49U17VismIhamnadWA7FEpkmvSV2zbFMT2sV3oNlT4jIMLicGJFp01ti19RleuqrpdZObEmDurnDUibB+dwSXMi9ZehwiKgVYWJHZNpMvkhRS62d2JIcrS3xaKequlgpmVyFgohaRoVag7xb5QB4jx2RqdJbYteUZXoaIi4uDoWFheLjypUrzXZsQ9IOx+7kcCwRtZCc4qr76yxlErSxtTJwNETUGHpL7JqyTE9DtMTaiYYwpEfVBIpDl/ORX1Ju4GiIqDUQS504WEMqlRg4GiJqDL0OxcbGxuLzzz/H2rVrkZmZiRkzZtRYpicuLk5sX15ejiNHjuDIkSMoLy/HtWvXcOTIEZw7d06fYRqldm1s0cPbERoB2HOKw7FEpH85LHVCZPL0uvJEQ5fpuX79Onr37i0+//DDD/Hhhx9i0KBBtVZ+N3dD/T2RmV2EXZlKjAluZ+hwiMjMKQo5cYLI1Ol9SbGGLNNTn2V4WpOhPTzxUcpZ/HomF2UValhbygwdEhGZMWWxdp1YJnZEpsrkZ8Was55tHeHpKEdpuRqpF24YOhwiMnMsdUJk+pjYGTGJRMJixUTUYrSJnZcT77EjMlVM7IzcUP/qxC5TyWFqItIr7XJing7ssSMyVUzsjFxYZ1fYWcmgLFLh+DXTX1WDiIyXsnryhAeHYolMFhM7Iye3kGFgV3cAwM6TzbcUGxHR3UpUlShWVQIAvJyY2BGZKiZ2JkBchYLLixGRnmhXnbCzksFerveCCUSkJ0zsTMDj3T0glQCZ2UW4erPU0OEQkRkSa9ixt47IpDGxMwEudlbo6+cCAEhhrx0R6UFOcXVix4kTRCaNiZ2JGNrjzuxYIqLmpuRyYkRmgYmdiQivLnuy/8INFJVVGDgaIjI3isLqUicciiUyaUzsTERHNzt0drdDhVrAr6dzDR0OEZkZJYdiicwCEzsTMtTfCwCHY4mo+Wlr2HE5MSLTxsTOhAz19wAA7DmVgwq1xsDREJE50fbYcTkxItPGxM6EBPm2gaudFYrKKnHwYr6hwyEiMyEIgricmAeHYolMGhM7EyKTSvBE96peu50cjiWiZlJQWoHyyqpRAA/OiiUyaUzsTMxQ/ztlTwRBMHA0RGQOtMOwLnZWkFvIDBwNETUFEzsT07+LG+QWUlzJv40zyluGDoeIzMCdYVj21hGZOiZ2JsbWygL9H3IDAOw8qTBwNERkDrQzYr1Yw47I5DGxM0HaYsU7ubwYETUDcdUJTpwgMnlM7EzQkOoJFH9dKUBO9QWZiKixFFxOjMhsMLEzQR6O1gjydQYApJxirx0RNY32HjsuJ0Zk+pjYmShxduxJlj0hoqbJ4XJiRGaDiZ2JCu9RldjtO5eH0vJKA0dDRKZMwckTRGaDiZ2J6uppD18XG6gqNfj9bJ6hwyEiE1Wp1iDvVnW5E95jR2TymNiZKIlEgqE9vABwOJaIGi/vVjk0QtXKNq52TOyITB0TOxMW7l81O3b3qRyoNVyFgogaTlvqxMNBDplUYuBoiKipmNiZsEf8XOBobYEbJeU4nHXT0OEQkQkSEztH3l9HZA6Y2JkwS5kUj1fXtNuZyeFYImo4bWLnxfvriMwCEzsTx7InRNQUYg079tgRmQUmdiZuYFd3WMokOJ9bggu5twwdDhGZGHE5MSZ2RGaBiZ2Jc7S2xKOdXAEAKVw7logaSMHEjsisMLEzA9pixTs5HEtEDZQjDsXyHjsic8DEzgwM6VE1geLQ5Xzkl5QbOBoiMiXssSMyL0zszEC7Nrbw93aERgBSODuWqFFWrlwJPz8/WFtbIzQ0FGlpaXW2HTx4MCQSSY3HiBEjxDaCIGDhwoXw9vaGjY0NwsPDcfbsWZ3j5OfnY+LEiXB0dISzszOmTJmCW7da7l7Zsgo1Cm9XAGBiR2QumNiZiWE9q1ahWL7rLIrKKgwcDZFpSU5ORmxsLOLj45GRkYHAwEBEREQgJ6f2+1a3bNmC7Oxs8XH8+HHIZDKMGzdObPP+++/jo48+wqpVq3DgwAHY2dkhIiICZWVlYpuJEyfixIkT2LlzJ7Zt24bffvsN06dP1/v5ammHYa0tpXC0tmix9yUi/WFiZyZe7N8R7V1sca3gNt7edtLQ4RCZlGXLlmHatGmIjo6Gv78/Vq1aBVtbW6xevbrW9i4uLvDy8hIfO3fuhK2trZjYCYKA5cuX480338Tf/vY39OrVC1999RWuX7+O7777DgCQmZmJHTt24IsvvkBoaCj69++PFStWICkpCdevX2+R81aINeysIZFw1Qkic9AiiV1DhjgAYNOmTejevTusra0REBCAH3/8sSXCNGl2cgt8OC4QEgmw8dBV1rUjqqfy8nKkp6cjPDxc3CaVShEeHo7U1NR6HSMxMRHjx4+HnZ0dAODixYtQKBQ6x3RyckJoaKh4zNTUVDg7O6Nv375im/DwcEilUhw4cKA5Tu2BuOoEkfnRe2LX0CGOP//8ExMmTMCUKVNw+PBhjBo1CqNGjcLx48f1HarJC+nogmkDOgEA5m85hpucSEH0QHl5eVCr1fD09NTZ7unpCYVC8cD909LScPz4cUydOlXcpt3vfsdUKBTw8PDQed3CwgIuLi51vq9KpUJRUZHOoylYw47I/Og9sWvoEMd//vMfPPXUU3jttdfQo0cPLFmyBH369MF///tffYdqFmKHdkUXD3vk3VLhze+ZDBPpW2JiIgICAhASEqL390pISICTk5P48PX1bdLxuJwYkfnRa2LXmCGO1NRUnfYAEBERUWf75v4Ga+qsLWVY9mwQZFIJth/Nxta/WuZeHSJT5ebmBplMBqVS9/YFpVIJLy+v++5bUlKCpKQkTJkyRWe7dr/7HdPLy6vGyEVlZSXy8/PrfN+4uDgUFhaKjytXrjz4BO+Dy4kRmR+9JnaNGeJQKBQNat/c32DNQUA7J8Q8/hAAYMH3x5FTVPaAPYhaLysrKwQHByMlJUXcptFokJKSgrCwsPvuu2nTJqhUKkyaNElne8eOHeHl5aVzzKKiIhw4cEA8ZlhYGAoKCpCeni622b17NzQaDUJDQ2t9P7lcDkdHR51HUyh4jx2R2TH5WbHN/Q3WXMQ88RB6tnVEQWkF5m85BkEQDB0SkdGKjY3F559/jrVr1yIzMxMzZsxASUkJoqOjAQBRUVGIi4ursV9iYiJGjRoFV1dXne0SiQRz5szB22+/jR9++AHHjh1DVFQUfHx8MGrUKABAjx498NRTT2HatGlIS0vDH3/8gZiYGIwfPx4+Pj56P2cA4pc+LyZ2RGZDr4WLGjPE4eXl1aD2crkccjnvD7mXpUyKZc8G4ekV+7D7VA42HrqCyEfaGzosIqMUGRmJ3NxcLFy4EAqFAkFBQdixY4c4epCVlQWpVPd78OnTp7Fv3z788ssvtR7zn//8J0pKSjB9+nQUFBSgf//+2LFjB6yt7yRR69evR0xMDIYMGQKpVIoxY8bgo48+0t+J3kUQhLuGYnkNJTIXEkHPXTmhoaEICQnBihUrAFQNcbRv3x4xMTGYP39+jfaRkZEoLS3F1q1bxW2PPfYYevXqhVWrVj3w/YqKiuDk5ITCwsImD1OYg89+O493fjwFOysZdswZCF8XW0OHRNRsWvvnvSnnX3i7AoGLq5LSU0uegrWlTB8hElEzaMhnXe9DsQ0d4pg9ezZ27NiBf/3rXzh16hQWLVqEQ4cOISYmRt+hmqUp/TvhEb82KClX47XNf0Gj4ZAsEd0ZhnWysWRSR2RG9J7YRUZG4sMPP8TChQsRFBSEI0eO1BjiyM7OFts/9thj2LBhAz777DMEBgZi8+bN+O6779CzZ099h2qWZFIJPhwXCBtLGfZfyMfa1EuGDomIjACHYYnMk96HYltaax+aqcvX+y9jwXfHIbeQ4sfZA9DZ3d7QIRE1WWv/vDfl/DenX8Wrm/7CgC5u+HpK7bNwicg4GNVQLBmHSaHtMaCLG1SVGvxj41+oVGsMHRIRGRBXnSAyT0zsWgmJRIL3xvSCg7UFjlwpwKe/XTB0SERkQEqWOiEyS0zsWhEfZxssGvkwAGD5rjM4eb11r9JB1Jrd6bHjPXZE5oSJXSvzTJ+2eNLfExVqAbEbj0BVqTZ0SERkANrJE1x1gsi8MLFrZSQSCd55JgAudlY4pSjGRylnDR0SERkAh2KJzBMTu1bIzV6Od0ZXlY/5ZO95ZGTdNHBERNSSNBoBOcXacidM7IjMCRO7Vuqpnt4YFeQDjQC8uvEv3C7nkCxRa5FXooJaI0AqAdzsrQwdDhE1IyZ2rdji/+sJT0c5LuSV4L0dpwwdDhG1kJzq++vc7OWwkPG/ASJzwk90K+Zka4n3xvQCAHz55yX8eT7PwBERUUtgDTsi88XErpUb3M0Dz4W2BwC8tukoissqDBwREembgokdkdliYkd4fXgP+LrY4FrBbby9LdPQ4RCRnnGdWCLzxcSOYC+3wIdjAyGRAMmHrmD3KaWhQyIiPcphjx2R2WJiRwCA0E6umNKvIwBg3jfHcLOk3MAREZG+KFjDjshsMbEj0asR3fCQhz1yi1VY+MMJQ4dDRHpyZ9UJDsUSmRsmdiSytpRh2bOBkEkl2PrXdWw7et3QIRGRHoirTjixx47I3DCxIx292jlj1uMPAQDe/O44corLDBwRETUnVaUa+dW3Wng6MLEjMjdM7KiGmMcfwsM+jigorUDcN8cgCIKhQyKiZpJbvZSYlYUUzraWBo6GiJobEzuqwcpCimXPBsFKJkXKqRxsSr9q6JCIqJncKU4sh0QiMXA0RNTcmNhRrbp5OSD2ya4AgLe2nsTVm6UGjoiImoNYw47DsERmiYkd1WnagE4I7tAGt1SV+Ofmo9BoOCRLZOq4nBiReWNiR3WSSSX417hA2FjK8Of5G/gq9ZKhQyIzU6KqxKW8Ehy8lI8fj2Vj7Z+X8OHPp1FYyqXt9IXLiRGZNwtDB0DGzc/NDq8P744F35/AuztOYWBXd3Rytzd0WGTEyis1yLulQm5x9ePuvxerql6r3lZarq71GMMDvOHEG/v1IofLiRGZNSZ29EATQzvg5xNK7DuXh39s+gubXgqDhYydva1N4e0KXC+4XWfCpn1eeLthvW22VjK4O8jhbi+v+tNBDns5L036oihkDTsic8arJz2QVCrB+2N7IeLfv+FwVgGW7TyDfzzZDTIpZ9SZu7xbKuw4rsD2o9k4cPEG6nubpaVMAnd7OdzuSdjuTeDc7OWwYxLXopTVtSk9OHmCyCzxikr14uNsg/j/exivbvoLH+89jx3HFXhlSBeMDPRhgmdmbtxSYceJqmRu/wXdZM7N3gpu9jUTtHufO9lYspSGkeJQLJF5Y2JH9TamT1sU3q7Ait1ncSGvBHOSj2DF7rN4ZUgXPN2LCZ4pyy8px8/VyVzqhRtQ35XN9WrnhBEB3hge4A1fF1sDRklNdUtViVuqSgCcPEFkrpjYUb1JJBJM6d8RkY/4Yu2fl/DZbxdwPrcEs5OOYMXuc5g9pAtGBHhDygTPJNzUJnPHsvHned1kLqCtE0b08sbwnt5o78pkzlxoS504yC04BE5kpvjJpgazl1tg1uMPISqsA7784xI+//0CzuXcwsv/O4wVu89i9pCuGNbTiwmeESooLccvJ5TYdiwbf57LQ+VdydzDPo4Y0csbIwK80cHVzoBRkr5oEzsPDsMSmS0mdtRoDtaWeHlIF0zu54c1+y7hi30XcEZ5C7M2ZKC7lwNmD+mCiIeZ4BlaYWkFfjlZ1TO376xuMtfD2xFP96oaZu3oxmTO3GkTO86IJTJfTOyoyRytLTE7vAte6OeH1fsuYvW+izilKMaM9VUJ3pzwLnjSnwleSyq8XYGdJ5XYfvQ69p3LQ4X6TjLX3ctBTOZYk7B14XJiROaPiR01GycbS8wd2hUv9uuIxH0XsPqPSzilKMbf12XA39sRs8O74El/T86W1JOisgrsOqnE9qPZ+O1srk4y183ToeqeuQBvPOTBZK610taw82SPHZHZYmJHzc7J1hKxT3bDi/074ovfL2LNHxdxMrsIL32djod9HDEnvCvCe3gwwWsGN26psPtUDn4+ocRvZ3JRrtaIr3X1tMeIAB+M6OWFhzwcDBglGYuc6hp2ng68x47IXDGxI71xtrXCqxHdMKV/R3z++wWs/fMSTlwvwrSvDiGgrRPmhHfBE92Z4DWEIAg4n1uCXZlK7DqpRHrWTQh31Zl7yMMeIwK8MaKXN7p6MpkjXeJQLEudEJktJnakd23srPDPp7pj6oBO+Oy3C/gq9RKOXSvElLWHENjOCXPCu2JwN3cmeHWoVGuQfvlmVTKXmYOLeSU6rz/s44jwHp4YHuCNrp72/DlSnTgUS2T+mNhRi3Gxs8L8Yd0xbUBHfPb7BXz152X8dbUQ0V8eRJCvM+aEd8GgrkzwgKpCsr+dycWuk0rsPp2DgtI7669ayiQI6+yGoT08MKSHJ3ycbQwYKZkKQRDuDMWyx47IbOktscvPz8fLL7+MrVu3QiqVYsyYMfjPf/4De/u6b9z+7LPPsGHDBmRkZKC4uBg3b96Es7OzvkIkA3G1lyNuWA9Mu6sH78iVAryw5iB6t3fG3PCuGNDFrdUleNcLbiMlU4mdmTnYf/6Gzv1yzraWeKKbB8L9PTGgixscrC0NGCmZopulFeKEGg/eY0dktvSW2E2cOBHZ2dnYuXMnKioqEB0djenTp2PDhg117lNaWoqnnnoKTz31FOLi4vQVGhkJN3s5Xh9eleB9+ut5fL3/Mg5nFSBqdRoC2zlhQBd39OngjN6+bdDGzsrQ4TY7QRBw4noRdp5UYlemEieuF+m87udqi6H+ngjv4YngDm1gIZMaKFIyB9phWDd7K1jy3xKR2dJLYpeZmYkdO3bg4MGD6Nu3LwBgxYoVGD58OD788EP4+PjUut+cOXMAAHv37tVHWGSk3B3kePNpf0wf1Amr9l7A+gNVQ7R/XS0U23Rys0Pv9m3ERK+bl4NJrk2rqlQj9fwN7MpUIiUzB9nV/9kCgEQCBLdvg/DqZK6zu12r67Uk/VFWD8N6sIYdkVnTS2KXmpoKZ2dnMakDgPDwcEilUhw4cACjR49utvdSqVRQqVTi86Kiovu0JmPm4WCNhSP98fdBnbAzU4mMywU4nHUTF/JKxMc3GVcBAHZWMgT6OqPPXcmesfbq5ZeUY8+pHOzKrCpJUlKuFl+zsZRhYFc3hPfwxBPdPeBqzyEy0g9lIVedIGoN9JLYKRQKeHh46L6RhQVcXFygUCia9b0SEhKwePHiZj0mGZaHozUmhnbAxNAOAKoWqz9ypQAZWTeRkXUTR7IKUFKuxp/nb+DP8zfE/Tq52SGofXWy175levUEQUBRWSWURWXILiyDovA2FIUqKIpuVz8vwxllMe5axQuejnKE96jqlQvr7AprS5leYyQC7i51wi8PROasQYnd/Pnz8d577923TWZmZpMCaqi4uDjExsaKz4uKiuDr69uiMZB+tbGzwuPdPfB496ovC2qNgDPKYhzOupPsXci906u3JeMaAN1evd7tndG7fRu4NKBXT6MRcKOkHIrCMiiKqpK2bPHvd/4svasHri7+3o4I9/fE0B6e6NnWkUOs1OI4FEvUOjQosfvHP/6BF1544b5tOnXqBC8vL+Tk5Ohsr6ysRH5+Pry8vBoc5P3I5XLI5fwG2prIpBL08HZED29HPBfaHoBur97hrAIcuVKAW6rKGr16Hd3s0Lu6Vy+wnTPK1WooClXILrytk6xlF5Yhp7hMZ1mu+3G2tYSXozW8nKzh7WQNL0cbeDnJ4eVkgy4e9ixJQgbHoVii1qFBiZ27uzvc3d0f2C4sLAwFBQVIT09HcHAwAGD37t3QaDQIDQ1tXKRE91Fbr97ZnGJkXNbt1buYV/XQ9uo9iEQCuNvL4e1kDU/H6qTNqTppc7QRt9tYcTjVlK1cuRIffPABFAoFAgMDsWLFCoSEhNTZvqCgAG+88Qa2bNmC/Px8dOjQAcuXL8fw4cMBAH5+frh8+XKN/WbOnImVK1cCAAYPHoxff/1V5/WXXnoJq1atasYzu0Mp1rDjF2Eic6aXe+x69OiBp556CtOmTcOqVatQUVGBmJgYjB8/XpwRe+3aNQwZMgRfffWVeAFVKBRQKBQ4d+4cAODYsWNwcHBA+/bt4eLioo9QyUzJpBJ093JEd687vXoFpeU4fKUAhy/fREZWAU5cL4Sd3KJm0nZXz5u7g5ylIcxccnIyYmNjsWrVKoSGhmL58uWIiIjA6dOna9wrDADl5eUYOnQoPDw8sHnzZrRt2xaXL1/Wqbl58OBBqNV3huiPHz+OoUOHYty4cTrHmjZtGt566y3xua2tbfOfYDVBqPpcsDgxkZkT9OTGjRvChAkTBHt7e8HR0VGIjo4WiouLxdcvXrwoABD27NkjbouPjxcA1HisWbOm3u9bWFgoABAKCwub8WyIyBg1x+c9JCREmDVrlvhcrVYLPj4+QkJCQq3tP/nkE6FTp05CeXl5vd9j9uzZQufOnQWNRiNuGzRokDB79uxGxy0IDT//SrVGUKs1D25IREalIZ91iSAI9buJyEQUFRXByckJhYWFcHR0NHQ4RKRHTf28l5eXw9bWFps3b8aoUaPE7ZMnT0ZBQQG+//77GvsMHz4cLi4usLW1xffffw93d3c899xzmDdvHmSymkPy5eXl8PHxQWxsLF5//XVx++DBg3HixAkIggAvLy+MHDkSCxYsuG+vXW3lnXx9fXm9IzJzDbnWca1YImq18vLyoFar4enpqbPd09MTp06dqnWfCxcuYPfu3Zg4cSJ+/PFHnDt3DjNnzkRFRQXi4+NrtP/uu+9QUFBQY+LZc889hw4dOsDHxwdHjx7FvHnzcPr0aWzZsqXOeFneiYgehIkdEVEDaDQaeHh44LPPPoNMJkNwcDCuXbuGDz74oNbELjExEcOGDaux4s706dPFvwcEBMDb2xtDhgzB+fPn0blz51rfm+WdiOhBmNgRUavl5uYGmUwGpVKps12pVNZZmsnb2xuWlpY6w649evSAQqFAeXk5rKzu1Eq8fPkydu3add9eOC1txYBz587VmdixvBMRPQin+xFRq2VlZYXg4GCkpKSI2zQaDVJSUhAWFlbrPv369cO5c+eg0WjEbWfOnIG3t7dOUgcAa9asgYeHB0aMGPHAWI4cOQKgKnEkImosJnZE1KrFxsbi888/x9q1a5GZmYkZM2agpKQE0dHRAICoqCjExcWJ7WfMmIH8/HzMnj0bZ86cwfbt2/HOO+9g1qxZOsfVaDRYs2YNJk+eDAsL3cGR8+fPY8mSJUhPT8elS5fwww8/ICoqCgMHDkSvXr30f9JEZLY4FEtErVpkZCRyc3OxcOFCKBQKBAUFYceOHeKEiqysLEild74D+/r64ueff8bcuXPRq1cvtG3bFrNnz8a8efN0jrtr1y5kZWXhxRdfrPGeVlZW2LVrF5YvX46SkhL4+vpizJgxePPNN/V7skRk9ljuhIhMVmv/vLf28ydqLRryWedQLBEREZGZYGJHREREZCaY2BERERGZCSZ2RERERGaCiR0RERGRmWBiR0RERGQmmNgRERERmQmzK1CsLctXVFRk4EiISN+0n3MzK8dZb7zeEbUODbnWmV1iV1xcDKCqOjwRtQ7FxcVwcnIydBgtjtc7otalPtc6s1t5QqPR4Pr163BwcIBEInlg+6KiIvj6+uLKlSsmX7ndXM7FXM4D4LnomyAIKC4uho+Pj86yX61FQ653xvj7ayyei/Exl/MAjPNcGnKtM7seO6lUinbt2jV4P0dHR6P5BTaVuZyLuZwHwHPRp9bYU6fVmOudsf3+moLnYnzM5TwA4zuX+l7rWt9XXCIiIiIzxcSOiIiIyEy0+sROLpcjPj4ecrnc0KE0mbmci7mcB8BzIeNhTr8/novxMZfzAEz/XMxu8gQRERFRa9Xqe+yIiIiIzAUTOyIiIiIzwcSOiIiIyEwwsSMiIiIyE606sVu5ciX8/PxgbW2N0NBQpKWlGTqkBktISMAjjzwCBwcHeHh4YNSoUTh9+rShw2oW7777LiQSCebMmWPoUBrl2rVrmDRpElxdXWFjY4OAgAAcOnTI0GE1iFqtxoIFC9CxY0fY2Nigc+fOWLJkSatdm9WU8XpnvHitMw7mcr1rtYldcnIyYmNjER8fj4yMDAQGBiIiIgI5OTmGDq1Bfv31V8yaNQv79+/Hzp07UVFRgSeffBIlJSWGDq1JDh48iE8//RS9evUydCiNcvPmTfTr1w+Wlpb46aefcPLkSfzrX/9CmzZtDB1ag7z33nv45JNP8N///heZmZl477338P7772PFihWGDo0agNc748VrnfEwm+ud0EqFhIQIs2bNEp+r1WrBx8dHSEhIMGBUTZeTkyMAEH799VdDh9JoxcXFQpcuXYSdO3cKgwYNEmbPnm3okBps3rx5Qv/+/Q0dRpONGDFCePHFF3W2PfPMM8LEiRMNFBE1Bq93xonXOuNiLte7VtljV15ejvT0dISHh4vbpFIpwsPDkZqaasDImq6wsBAA4OLiYuBIGm/WrFkYMWKEzu/H1Pzwww/o27cvxo0bBw8PD/Tu3Ruff/65ocNqsMceewwpKSk4c+YMAOCvv/7Cvn37MGzYMANHRvXF653x4rXOuJjL9c7C0AEYQl5eHtRqNTw9PXW2e3p64tSpUwaKquk0Gg3mzJmDfv36oWfPnoYOp1GSkpKQkZGBgwcPGjqUJrlw4QI++eQTxMbG4vXXX8fBgwfxyiuvwMrKCpMnTzZ0ePU2f/58FBUVoXv37pDJZFCr1Vi6dCkmTpxo6NConni9M0681hkfc7netcrEzlzNmjULx48fx759+wwdSqNcuXIFs2fPxs6dO2FtbW3ocJpEo9Ggb9++eOeddwAAvXv3xvHjx7Fq1SqTutht3LgR69evx4YNG/Dwww/jyJEjmDNnDnx8fEzqPMj8mPL1jtc642Qu17tWmdi5ublBJpNBqVTqbFcqlfDy8jJQVE0TExODbdu24bfffkO7du0MHU6jpKenIycnB3369BG3qdVq/Pbbb/jvf/8LlUoFmUxmwAjrz9vbG/7+/jrbevTogW+++cZAETXOa6+9hvnz52P8+PEAgICAAFy+fBkJCQkmdaFrzXi9Mz681hknc7netcp77KysrBAcHIyUlBRxm0ajQUpKCsLCwgwYWcMJgoCYmBh8++232L17Nzp27GjokBptyJAhOHbsGI4cOSI++vbti4kTJ+LIkSMmc6EDgH79+tUow3DmzBl06NDBQBE1TmlpKaRS3cuETCaDRqMxUETUULzeGR9e64yT2VzvDD17w1CSkpIEuVwufPnll8LJkyeF6dOnC87OzoJCoTB0aA0yY8YMwcnJSdi7d6+QnZ0tPkpLSw0dWrMw1ZliaWlpgoWFhbB06VLh7Nmzwvr16wVbW1th3bp1hg6tQSZPniy0bdtW2LZtm3Dx4kVhy5Ytgpubm/DPf/7T0KFRA/B6Z/x4rTM8c7netdrEThAEYcWKFUL79u0FKysrISQkRNi/f7+hQ2owALU+1qxZY+jQmoWpXuwEQRC2bt0q9OzZU5DL5UL37t2Fzz77zNAhNVhRUZEwe/ZsoX379oK1tbXQqVMn4Y033hBUKpWhQ6MG4vXOuPFaZ3jmcr2TCIKJlVQmIiIiolq1ynvsiIiIiMwREzsiIiIiM8HEjoiIiMhMMLEjIiIiMhNM7IiIiIjMBBM7IiIiIjPBxI6IiIjITDCxIyIiIjITTOyIiIiIzAQTOyIiIiIzwcSOiIiIyEwwsSMiIiIyE/8PvMw2fj/hepwAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, axs = plt.subplots(1, 2)\n", + "axs[0].set_title(\"Log loss value on eval set\")\n", + "axs[0].plot(np.log(eval_metrics_history[\"test_loss\"]))\n", + "axs[1].set_title(\"Accuracy on eval set\")\n", + "axs[1].plot(eval_metrics_history[\"test_accuracy\"])\n", + "plt.tight_layout();" + ] + }, + { + "cell_type": "markdown", + "id": "a3f7b0ad-ddfa-4ab3-b56f-6ea99385ff6a", + "metadata": {}, + "source": [ + "## Use Model for Inference\n", + "After all that, the product of what we were working for: a trained model we can save and load for inference. For people using LLMs recently, this pattern may look rather familiar: an input sentence tokenized into an array and computed 'next' token-by-token. While many recent LLMs are decoder-only, this was an encoder/decoder architecture with the very specific english-to-spanish pattern baked in.\n", + "\n", + "We've changed a couple things from the source 'use' function, here - because of the tokenizer used, things like `[start]` and `[end]` are no longer single tokens - instead `[start]` is `[29563, 60] = \"[start\" + \"]\"` and `[end]` is `[58308, 60] = \"[end\" + \"]\"` - thus we start with only a single token `[start` and can't only test on `last_token = \"[end\"]`. Otherwise, the main change here is that the input is assumed a single sentence, rather than batch inference." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "e4589706-cfd6-4efb-9975-bfa0df75d4f0", + "metadata": {}, + "outputs": [], + "source": [ + "def decode_sequence(input_sentence):\n", + "\n", + " input_sentence = custom_standardization(input_sentence)\n", + " tokenized_input_sentence = tokenize_and_pad(input_sentence, tokenizer, sequence_length)\n", + "\n", + " decoded_sentence = \"[start\"\n", + " for i in range(sequence_length):\n", + " tokenized_target_sentence = tokenize_and_pad(decoded_sentence, tokenizer, sequence_length)[:-1]\n", + " predictions = eval_model(jnp.array([tokenized_input_sentence]), jnp.array([tokenized_target_sentence]))\n", + "\n", + " sampled_token_index = np.argmax(predictions[0,i, :]).item(0)\n", + " sampled_token = tokenizer.decode([sampled_token_index])\n", + " decoded_sentence += \"\" + sampled_token\n", + "\n", + " if decoded_sentence[-5:] == \"[end]\":\n", + " break\n", + " return decoded_sentence" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "554c2f72-0bd3-4ed1-804b-5f1a4cc13851", + "metadata": {}, + "outputs": [], + "source": [ + "test_eng_texts = [pair[0] for pair in test_pairs]" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "c1d6edbb-af89-42c9-90c3-d61612b75da3", + "metadata": {}, + "outputs": [], + "source": [ + "test_result_pairs = []\n", + "for _ in range(10):\n", + " input_sentence = random.choice(test_eng_texts)\n", + " translated = decode_sequence(input_sentence)\n", + "\n", + " test_result_pairs.append(f\"[Input]: {input_sentence} [Translation]: {translated}\")" + ] + }, + { + "cell_type": "markdown", + "id": "258c2172-5a0f-4dee-9b21-f65433183c62", + "metadata": {}, + "source": [ + "## Test Results\n", + "For the model and the data, not too shabby - It's definitely spanish-ish. Though when 'making' friends, please don't confuse 'hacer' (to make) with 'comer' (to eat)." + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "4f0ae018-b7cd-4849-b245-c5c647ad1a95", + "metadata": { + "outputId": "c35060a1-a0c6-43ad-e21c-b4dbff8e2536" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[Input]: Please feel free to ask questions. [Translation]: [start] por favor sienten libre preguntas [end]\n", + "[Input]: I'll be back late tonight. [Translation]: [start] volveré a tarde de esta noche [end]\n", + "[Input]: She suggested that the customer buy a blue tie. [Translation]: [start] ella sugirió el cliente que comprara un cliente azul rico [end]\n", + "[Input]: Where's the toothpaste? [Translation]: [start] dónde está la pasta de dientes [end]\n", + "[Input]: A lot of soldiers were killed here. [Translation]: [start] muchos soldados de fallecieron aquí [end]\n", + "[Input]: I think the eggs that I just ate were rotten. [Translation]: [start] creo que los huevos que me me estaban podían hacer [end]\n", + "[Input]: Are you in a hurry? [Translation]: [start] estás apurado [end]\n", + "[Input]: Are we going for a walk? [Translation]: [start] vamos a ir a dar un paseo [end]\n", + "[Input]: We can still break even. [Translation]: [start] podemos seguir quieto quieto [end]\n", + "[Input]: I don't know where it is. [Translation]: [start] no sé dónde está [end]\n" + ] + } + ], + "source": [ + "for i in test_result_pairs:\n", + " print(i)" + ] + }, + { + "cell_type": "markdown", + "id": "5ca18d4c-b3c0-4abb-b5fc-fc96a2264b53", + "metadata": {}, + "source": [ + "Example output from the above cell:\n", + "\n", + " [Input]: We're going to have a baby. [Translation]: [start] nosotros vamos a tener un bebé [end]\n", + " [Input]: You drive too fast. [Translation]: [start] conducís demasiado rápido [end]\n", + " [Input]: Let me know if there's anything I can do. [Translation]: [start] déjame saber si hay cualquier cosa que yo pueda hacer [end]\n", + " [Input]: Let's go to the kitchen. [Translation]: [start] vayamos a la cocina [end]\n", + " [Input]: Tom gasped. [Translation]: [start] tom se quedó sin aliento [end]\n", + " [Input]: I was just hanging out with some of my friends. [Translation]: [start] estaba escquieto con algunos de mi amigos [end]\n", + " [Input]: Tom is in the bathroom. [Translation]: [start] tom está en el cuarto de baño [end]\n", + " [Input]: I feel safe here. [Translation]: [start] me siento segura [end]\n", + " [Input]: I'm going to need you later. [Translation]: [start] me voy a necesitar después [end]\n", + " [Input]: A party is a good place to make friends with other people. [Translation]: [start] una fiesta es un buen lugar de comer amigos con otras personas [end]" + ] + } + ], + "metadata": { + "accelerator": "TPU", + "jupytext": { + "formats": "ipynb,md", + "main_language": "python" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs_nnx/examples/machine_translation.md b/docs_nnx/examples/machine_translation.md new file mode 100644 index 000000000..b5c6ab0be --- /dev/null +++ b/docs_nnx/examples/machine_translation.md @@ -0,0 +1,580 @@ +--- +jupyter: + jupytext: + formats: ipynb,md + main_language: python + text_representation: + extension: .md + format_name: markdown + format_version: '1.3' + jupytext_version: 1.13.8 +--- + +# Machine Translation with encoder-decoder transformer model + + +This tutorial is adapted from [Keras' documentation on English-to-Spanish translation with a sequence-to-sequence Transformer](https://keras.io/examples/nlp/neural_machine_translation_with_transformer/), which is itself an adaptation from the book [Deep Learning with Python, Second Edition by François Chollet](https://www.manning.com/books/deep-learning-with-python-second-edition) + +We step through an encoder-decoder transformer in JAX and train a model for English->Spanish translation. + +```python +# !pip install tiktoken grain flax optax +``` + +Standard library and data handling imports: + +```python +import pathlib +import random +import string +import re +import numpy as np +``` + +JAX, Flax, and training framework imports: + +```python +import jax.numpy as jnp +import optax + +from flax import nnx +``` + +Tokenizer (`tiktoken`), data loader (`grain`), and progress bar (`tqdm`): + +```python +import tiktoken +import grain.python as grain +import tqdm +``` + +## Pull down data to temp and extract into memory + +There are lots of ways to get this done, but for simplicity and clear visibility into what's happening this is downloaded to a temporary directory, extracted there, and read into a python object with processing. + +```python +import requests +import zipfile +import tempfile + +url = "http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip" + +with tempfile.TemporaryDirectory() as temp_dir: + temp_path = pathlib.Path(temp_dir) + zip_file_path = temp_path / "spa-eng.zip" + + response = requests.get(url) + zip_file_path.write_bytes(response.content) + + with zipfile.ZipFile(zip_file_path, "r") as zip_ref: + zip_ref.extractall(temp_path) + + text_file = temp_path / "spa-eng" / "spa.txt" + + with open(text_file) as f: + lines = f.read().split("\n")[:-1] + text_pairs = [] + for line in lines: + eng, spa = line.split("\t") + spa = "[start] " + spa + " [end]" + text_pairs.append((eng, spa)) +``` + +## Build train/validate/test pair sets +We'll stay close to the original tutorial so it's clear how to follow what's the same vs what's different; one early difference is the choice to go with an off-the-shelf encoder/tokenizer in tiktoken. Specifically "cl100k_base" - it has a wide range of language understanding and it's fast. + +```python outputId="cf7fc8d5-029d-48d6-d739-95c53bdc9b38" +random.shuffle(text_pairs) +num_val_samples = int(0.15 * len(text_pairs)) +num_train_samples = len(text_pairs) - 2 * num_val_samples +train_pairs = text_pairs[:num_train_samples] +val_pairs = text_pairs[num_train_samples : num_train_samples + num_val_samples] +test_pairs = text_pairs[num_train_samples + num_val_samples :] + +print(f"{len(text_pairs)} total pairs") +print(f"{len(train_pairs)} training pairs") +print(f"{len(val_pairs)} validation pairs") +print(f"{len(test_pairs)} test pairs") +``` + +```python +tokenizer = tiktoken.get_encoding("cl100k_base") +``` + +We strip out punctuation to keep things simple and in line with the original tutorial - the `[` `]` are kept in so that our `[start]` and `[end]` formatting is preserved. We also record the vocabulary size from the tokenizer and fix the maximum sequence length for all inputs. + +```python +strip_chars = string.punctuation + "¿" +strip_chars = strip_chars.replace("[", "") +strip_chars = strip_chars.replace("]", "") + +vocab_size = tokenizer.n_vocab +sequence_length = 20 +``` + +`custom_standardization` lowercases a string and removes the punctuation characters defined above: + +```python +def custom_standardization(input_string): + lowercase = input_string.lower() + return re.sub(f"[{re.escape(strip_chars)}]", "", lowercase) +``` + +`tokenize_and_pad` encodes a string into token IDs, truncates to `max_length` if needed, and zero-pads shorter sequences so every example is the same fixed length: + +```python +def tokenize_and_pad(text, tokenizer, max_length): + tokens = tokenizer.encode(text)[:max_length] + padded = tokens + [0] * (max_length - len(tokens)) if len(tokens) < max_length else tokens ##assumes list-like - (https://github.com/openai/tiktoken/blob/main/tiktoken/core.py#L81 current tiktoken out) + return padded +``` + +`format_dataset` standardizes and tokenizes both the English and Spanish strings, then returns three arrays: +- `encoder_inputs` — the full tokenized English sentence. +- `decoder_inputs` — the Spanish sentence shifted right by one (all tokens except the last), used as the decoder prompt at each step. +- `target_output` — the Spanish sentence shifted left by one (all tokens except the first), used as the prediction target. + +```python +def format_dataset(eng, spa, tokenizer, sequence_length): + eng = custom_standardization(eng) + spa = custom_standardization(spa) + eng = tokenize_and_pad(eng, tokenizer, sequence_length) + spa = tokenize_and_pad(spa, tokenizer, sequence_length) + return { + "encoder_inputs": eng, + "decoder_inputs": spa[:-1], + "target_output": spa[1:], + } +``` + +Apply the preprocessing to every split to produce the final in-memory datasets: + +```python +train_data = [format_dataset(eng, spa, tokenizer, sequence_length) for eng, spa in train_pairs] +val_data = [format_dataset(eng, spa, tokenizer, sequence_length) for eng, spa in val_pairs] +test_data = [format_dataset(eng, spa, tokenizer, sequence_length) for eng, spa in test_pairs] +``` + +At this point we've extracted the data, applied formatting, and tokenized the phrases with padding. The data is kept in train/validate/test sets that each have dictionary entries, which look like the following: + +```python outputId="017a7188-ff72-4f53-b725-be16f48cdd9f" +## data selection example +print(train_data[135]) +``` + +The output should look something like + +{'encoder_inputs': [9514, 265, 3339, 264, 2466, 16930, 1618, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'decoder_inputs': [29563, 60, 1826, 7206, 71086, 37116, 653, 16109, 1493, 54189, 510, 408, 60, 0, 0, 0, 0, 0, 0], 'target_output': [60, 1826, 7206, 71086, 37116, 653, 16109, 1493, 54189, 510, 408, 60, 0, 0, 0, 0, 0, 0, 0]} + + +## Define Transformer components: Encoder, Decoder, Positional Embed + +In many ways this is very similar to the original source, with `ops` changing to `jnp` and `keras` or `layers` becoming `nnx`. Certain module-specific arguments come and go, like the rngs attached to most things in the updated version, and `decode=False` in the `MultiHeadAttention` call. + +`TransformerEncoder` implements a standard encoder block: self-attention over the input sequence followed by a two-layer feed-forward projection, with a residual connection and layer norm after each sub-layer. + +```python +class TransformerEncoder(nnx.Module): + def __init__(self, embed_dim: int, dense_dim: int, num_heads: int, rngs: nnx.Rngs, **kwargs): + self.embed_dim = embed_dim + self.dense_dim = dense_dim + self.num_heads = num_heads + + self.attention = nnx.MultiHeadAttention(num_heads=num_heads, + in_features=embed_dim, + decode=False, + rngs=rngs) + self.dense_proj = nnx.Sequential( + nnx.Linear(embed_dim, dense_dim, rngs=rngs), + nnx.relu, + nnx.Linear(dense_dim, embed_dim, rngs=rngs), + ) + + self.layernorm_1 = nnx.LayerNorm(embed_dim, rngs=rngs) + self.layernorm_2 = nnx.LayerNorm(embed_dim, rngs=rngs) + + def __call__(self, inputs): + attention_output = self.attention( + inputs_q = inputs, inputs_k = inputs, inputs_v = inputs, decode = False + ) + proj_input = self.layernorm_1(inputs + attention_output) + proj_output = self.dense_proj(proj_input) + return self.layernorm_2(proj_input + proj_output) +``` + +`PositionalEmbedding` combines two learned embedding tables: one maps token IDs to vectors, the other maps position indices (0, 1, 2, …) to vectors. Their sum gives each token both a semantic and a positional representation. `compute_mask` returns a boolean array that marks non-padding tokens (anything that is not token ID 0). + +```python +class PositionalEmbedding(nnx.Module): + def __init__(self, sequence_length: int, vocab_size: int, embed_dim: int, rngs: nnx.Rngs, **kwargs): + self.token_embeddings = nnx.Embed(num_embeddings=vocab_size, features=embed_dim, rngs=rngs) + self.position_embeddings = nnx.Embed(num_embeddings=sequence_length, features=embed_dim, rngs=rngs) + self.sequence_length = sequence_length + self.vocab_size = vocab_size + self.embed_dim = embed_dim + + def __call__(self, inputs): + length = inputs.shape[1] + positions = jnp.arange(0, length)[None, :] + embedded_tokens = self.token_embeddings(inputs) + embedded_positions = self.position_embeddings(positions) + return embedded_tokens + embedded_positions + + def compute_mask(self, inputs, mask=None): + if mask is None: + return None + else: + return jnp.not_equal(inputs, 0) +``` + +`TransformerDecoder` implements the decoder block with two attention layers: +- `attention_1` is masked self-attention over the target sequence. A causal mask prevents each position from attending to future tokens, which is essential so the model can only use tokens it has already generated when predicting the next one. +- `attention_2` is cross-attention where queries come from the decoder and keys/values come from the encoder output, letting the decoder attend to the full source sentence at every step. + +Each attention layer is followed by a residual connection, layer norm, and a shared feed-forward projection. + +```python +class TransformerDecoder(nnx.Module): + def __init__(self, embed_dim: int, latent_dim: int, num_heads: int, rngs: nnx.Rngs, **kwargs): + self.embed_dim = embed_dim + self.latent_dim = latent_dim + self.num_heads = num_heads + self.attention_1 = nnx.MultiHeadAttention(num_heads=num_heads, + in_features=embed_dim, + decode=False, + rngs=rngs) + self.attention_2 = nnx.MultiHeadAttention(num_heads=num_heads, + in_features=embed_dim, + decode=False, + rngs=rngs) + + self.dense_proj = nnx.Sequential( + nnx.Linear(embed_dim, latent_dim, rngs=rngs), + nnx.relu, + nnx.Linear(latent_dim, embed_dim, rngs=rngs), + ) + self.layernorm_1 = nnx.LayerNorm(embed_dim, rngs=rngs) + self.layernorm_2 = nnx.LayerNorm(embed_dim, rngs=rngs) + self.layernorm_3 = nnx.LayerNorm(embed_dim, rngs=rngs) + + def __call__(self, inputs, encoder_outputs): + causal_mask = nnx.make_causal_mask(inputs[:,:,0]) + attention_output_1 = self.attention_1( + inputs_q=inputs, inputs_v=inputs, inputs_k=inputs, mask=causal_mask + ) + out_1 = self.layernorm_1(inputs + attention_output_1) + + attention_output_2 = self.attention_2( + inputs_q=out_1, + inputs_v=encoder_outputs, + inputs_k=encoder_outputs + ) + out_2 = self.layernorm_2(out_1 + attention_output_2) + + proj_output = self.dense_proj(out_2) + return self.layernorm_3(out_2 + proj_output) +``` + +`TransformerModel` wires all the components together into the full encoder-decoder architecture. The same `positional_embedding` layer is reused for both the source and target sequences. The forward pass: +1. Embeds and positionally encodes the English (`encoder_inputs`) tokens and passes them through the encoder. +2. Embeds and positionally encodes the Spanish (`decoder_inputs`) tokens, passes them through the decoder along with the encoder output, and applies dropout. +3. Projects the decoder output to a distribution over the vocabulary with a final linear layer. + +```python +class TransformerModel(nnx.Module): + def __init__(self, sequence_length: int, vocab_size: int, embed_dim: int, latent_dim: int, num_heads: int, dropout_rate: float, rngs: nnx.Rngs): + self.sequence_length = sequence_length + self.vocab_size = vocab_size + self.embed_dim = embed_dim + self.latent_dim = latent_dim + self.num_heads = num_heads + self.dropout_rate = dropout_rate + + self.encoder = TransformerEncoder(embed_dim, latent_dim, num_heads, rngs=rngs) + self.positional_embedding = PositionalEmbedding(sequence_length, vocab_size, embed_dim, rngs=rngs) + self.decoder = TransformerDecoder(embed_dim, latent_dim, num_heads, rngs=rngs) + self.dropout = nnx.Dropout(rate=dropout_rate, rngs=rngs) + self.dense = nnx.Linear(embed_dim, vocab_size, rngs=rngs) + + def __call__(self, encoder_inputs: jnp.array, decoder_inputs: jnp.array): + x = self.positional_embedding(encoder_inputs) + encoder_outputs = self.encoder(x) + + x = self.positional_embedding(decoder_inputs) + decoder_outputs = self.decoder(x, encoder_outputs) + decoder_outputs = self.dropout(decoder_outputs, deterministic=False) + + logits = self.dense(decoder_outputs) + return logits +``` + +## Build out Data Loader and Training Definitions +It can be more computationally efficient to use pygrain for the data load stage, but this way it's abundandtly clear what's happening: data pairs go in and sets of jnp arrays come out, in step with our original dictionaries. 'Encoder_inputs', 'decoder_inputs' and 'target_output'. + +```python +batch_size = 512 #set here for the loader and model train later on + +class CustomPreprocessing(grain.MapTransform): + def __init__(self): + pass + + def map(self, data): + return { + "encoder_inputs": np.array(data["encoder_inputs"]), + "decoder_inputs": np.array(data["decoder_inputs"]), + "target_output": np.array(data["target_output"]), + } + +train_sampler = grain.IndexSampler( + len(train_data), + shuffle=True, + seed=12, # Seed for reproducibility + shard_options=grain.NoSharding(), # No sharding since it's a single-device setup + num_epochs=1, # Iterate over the dataset for one epoch +) + +val_sampler = grain.IndexSampler( + len(val_data), + shuffle=False, + seed=12, + shard_options=grain.NoSharding(), + num_epochs=1, +) + +train_loader = grain.DataLoader( + data_source=train_data, + sampler=train_sampler, # Sampler to determine how to access the data + worker_count=4, # Number of child processes launched to parallelize the transformations + worker_buffer_size=2, # Count of output batches to produce in advance per worker + operations=[ + CustomPreprocessing(), + grain.Batch(batch_size=batch_size, drop_remainder=True), + ] +) + +val_loader = grain.DataLoader( + data_source=val_data, + sampler=val_sampler, + worker_count=4, + worker_buffer_size=2, + operations=[ + CustomPreprocessing(), + grain.Batch(batch_size=batch_size), + ] +) +``` + +Optax doesn't have the identical loss function that the source tutorial uses, but this softmax cross entropy works well here - you can one_hot_encode if you don't use the `_with_integer_labels` version of the loss. + +```python +def compute_loss(logits, labels): + loss = optax.softmax_cross_entropy_with_integer_labels(logits=logits, labels=labels) + return jnp.mean(loss) +``` + +While in the original tutorial most of the model and training details happen inside keras, we make them explicit here in our step functions, which are later used in `train_one_epoch` and `evaluate_model`. + +`train_step` runs a single forward pass, computes the loss, calculates gradients with `nnx.value_and_grad`, and applies them via the optimizer. It is JIT-compiled for performance. + +```python +@nnx.jit +def train_step(model, optimizer, batch): + def loss_fn(model, train_encoder_input, train_decoder_input, train_target_input): + logits = model(train_encoder_input, train_decoder_input) + loss = compute_loss(logits, train_target_input) + return loss + + grad_fn = nnx.value_and_grad(loss_fn) + loss, grads = grad_fn(model, jnp.array(batch["encoder_inputs"]), jnp.array(batch["decoder_inputs"]), jnp.array(batch["target_output"])) + optimizer.update(grads) + return loss +``` + +`eval_step` runs a forward pass without updating weights and accumulates loss and accuracy into `eval_metrics`: + +```python +@nnx.jit +def eval_step(model, batch, eval_metrics): + logits = model(jnp.array(batch["encoder_inputs"]), jnp.array(batch["decoder_inputs"])) + loss = compute_loss(logits, jnp.array(batch["target_output"])) + labels = jnp.array(batch["target_output"]) + + eval_metrics.update( + loss=loss, + logits=logits, + labels=labels, + ) +``` + +`nnx.MultiMetric` accumulates loss and accuracy across batches. Separate history dictionaries record per-epoch values for later plotting: + +```python +eval_metrics = nnx.MultiMetric( + loss=nnx.metrics.Average('loss'), + accuracy=nnx.metrics.Accuracy(), +) + +train_metrics_history = { + "train_loss": [], +} + +eval_metrics_history = { + "test_loss": [], + "test_accuracy": [], +} +``` + +Key hyperparameters for the model and training run: +- `embed_dim` — dimensionality of token and position embeddings. +- `latent_dim` — width of the feed-forward projection inside each encoder/decoder block. +- `num_heads` — number of attention heads. +- `dropout_rate` — fraction of activations dropped during training. +- `learning_rate` / `num_epochs` — AdamW step size and training duration. + +```python +## Hyperparameters +rng = nnx.Rngs(0) +embed_dim = 256 +latent_dim = 2048 +num_heads = 8 +dropout_rate = 0.5 +vocab_size = tokenizer.n_vocab +sequence_length = 20 +learning_rate = 1.5e-3 +num_epochs = 10 +``` + +Although we'll want full dropout randomization for training, we'll want to evaluate our model without dropout by setting the `deterministic=True` flag. We'll make two views of our model (`train_model` and `eval_model`): one with each flag setting. Both views share the same underlying parameters, so a weight update through `train_model` is immediately visible when running `eval_model`. + +```python +model = TransformerModel(sequence_length, vocab_size, embed_dim, latent_dim, num_heads, dropout_rate, rngs=rng) +train_model = nnx.view(model, deterministic=False) +eval_model = nnx.view(model, deterministic=True) + +optimizer = nnx.ModelAndOptimizer(model, optax.adamw(learning_rate)) +``` + +`train_one_epoch` iterates over all training batches, calls `train_step` on each, and logs the per-step loss. `evaluate_model` resets the accumulated metrics, runs `eval_step` over the entire validation set, then prints and records the epoch-level loss and accuracy: + +```python +bar_format = "{desc}[{n_fmt}/{total_fmt}]{postfix} [{elapsed}<{remaining}]" +train_total_steps = len(train_data) // batch_size + +def train_one_epoch(epoch): + with tqdm.tqdm( + desc=f"[train] epoch: {epoch}/{num_epochs}, ", + total=train_total_steps, + bar_format=bar_format, + leave=True, + ) as pbar: + for batch in train_loader: + loss = train_step(train_model, optimizer, batch) + train_metrics_history["train_loss"].append(loss.item()) + pbar.set_postfix({"loss": loss.item()}) + pbar.update(1) + + +def evaluate_model(epoch): + # Compute the metrics on the train and val sets after each training epoch. + + + eval_metrics.reset() # Reset the eval metrics + for val_batch in val_loader: + eval_step(eval_model, val_batch, eval_metrics) + + for metric, value in eval_metrics.compute().items(): + eval_metrics_history[f'test_{metric}'].append(value) + + print(f"[test] epoch: {epoch + 1}/{num_epochs}") + print(f"- total loss: {eval_metrics_history['test_loss'][-1]:0.4f}") + print(f"- Accuracy: {eval_metrics_history['test_accuracy'][-1]:0.4f}") +``` + +## Start the Training! +With our data loaders set and the model, optimizer, and epoch train/eval functions set up - time to finally press go - on a 3090, this is roughly 19GB VRAM and each epoch is roughly 18 seconds with batch_size set to 512. + +```python outputId="a4f97db4-da4e-481b-f5c2-7137216e6380" +for epoch in range(num_epochs): + train_one_epoch(epoch) + evaluate_model(epoch) +``` + +We can then plot the loss over training time. That log-plot comes in handy here, or it's hard to appreciate the progress after 1000 steps or so. + +```python outputId="c1cdf060-335f-4321-8982-7ed07d15db06" +import matplotlib.pyplot as plt + +plt.plot(train_metrics_history["train_loss"], label="Loss value during the training") +plt.yscale('log') +plt.legend(); +``` + +And eval set Loss and Accuracy - Accuracy does continue to rise, though it's hard-earned progress after about the 5th epoch. Based on the training statistics, it's fair to say the process starts overfitting after roughly that 5th epoch. + +```python outputId="54680efe-bcf5-44df-e468-3054fb1e56bc" +fig, axs = plt.subplots(1, 2) +axs[0].set_title("Log loss value on eval set") +axs[0].plot(np.log(eval_metrics_history["test_loss"])) +axs[1].set_title("Accuracy on eval set") +axs[1].plot(eval_metrics_history["test_accuracy"]) +plt.tight_layout(); +``` + +## Use Model for Inference +After all that, the product of what we were working for: a trained model we can save and load for inference. For people using LLMs recently, this pattern may look rather familiar: an input sentence tokenized into an array and computed 'next' token-by-token. While many recent LLMs are decoder-only, this was an encoder/decoder architecture with the very specific english-to-spanish pattern baked in. + +We've changed a couple things from the source 'use' function, here - because of the tokenizer used, things like `[start]` and `[end]` are no longer single tokens - instead `[start]` is `[29563, 60] = "[start" + "]"` and `[end]` is `[58308, 60] = "[end" + "]"` - thus we start with only a single token `[start` and can't only test on `last_token = "[end"]`. Otherwise, the main change here is that the input is assumed a single sentence, rather than batch inference. + +```python +def decode_sequence(input_sentence): + + input_sentence = custom_standardization(input_sentence) + tokenized_input_sentence = tokenize_and_pad(input_sentence, tokenizer, sequence_length) + + decoded_sentence = "[start" + for i in range(sequence_length): + tokenized_target_sentence = tokenize_and_pad(decoded_sentence, tokenizer, sequence_length)[:-1] + predictions = eval_model(jnp.array([tokenized_input_sentence]), jnp.array([tokenized_target_sentence])) + + sampled_token_index = np.argmax(predictions[0,i, :]).item(0) + sampled_token = tokenizer.decode([sampled_token_index]) + decoded_sentence += "" + sampled_token + + if decoded_sentence[-5:] == "[end]": + break + return decoded_sentence +``` + +```python +test_eng_texts = [pair[0] for pair in test_pairs] +``` + +```python +test_result_pairs = [] +for _ in range(10): + input_sentence = random.choice(test_eng_texts) + translated = decode_sequence(input_sentence) + + test_result_pairs.append(f"[Input]: {input_sentence} [Translation]: {translated}") +``` + +## Test Results +For the model and the data, not too shabby - It's definitely spanish-ish. Though when 'making' friends, please don't confuse 'hacer' (to make) with 'comer' (to eat). + +```python outputId="c35060a1-a0c6-43ad-e21c-b4dbff8e2536" +for i in test_result_pairs: + print(i) +``` + +Example output from the above cell: + + [Input]: We're going to have a baby. [Translation]: [start] nosotros vamos a tener un bebé [end] + [Input]: You drive too fast. [Translation]: [start] conducís demasiado rápido [end] + [Input]: Let me know if there's anything I can do. [Translation]: [start] déjame saber si hay cualquier cosa que yo pueda hacer [end] + [Input]: Let's go to the kitchen. [Translation]: [start] vayamos a la cocina [end] + [Input]: Tom gasped. [Translation]: [start] tom se quedó sin aliento [end] + [Input]: I was just hanging out with some of my friends. [Translation]: [start] estaba escquieto con algunos de mi amigos [end] + [Input]: Tom is in the bathroom. [Translation]: [start] tom está en el cuarto de baño [end] + [Input]: I feel safe here. [Translation]: [start] me siento segura [end] + [Input]: I'm going to need you later. [Translation]: [start] me voy a necesitar después [end] + [Input]: A party is a good place to make friends with other people. [Translation]: [start] una fiesta es un buen lugar de comer amigos con otras personas [end]