diff --git a/docs/cli/index.rst b/docs/cli/index.rst deleted file mode 100644 index 4fc4440881..0000000000 --- a/docs/cli/index.rst +++ /dev/null @@ -1,12 +0,0 @@ -Command-line interface -====================== - -.. click:: giskard.cli:cli - :prog: giskard - :nested: full - -.. toctree:: - :maxdepth: 1 - :hidden: - - ngrok/index \ No newline at end of file diff --git a/docs/cli/ngrok/index.rst b/docs/cli/ngrok/index.rst deleted file mode 100644 index b17dc7c887..0000000000 --- a/docs/cli/ngrok/index.rst +++ /dev/null @@ -1,43 +0,0 @@ -Setup a :code:`ngrok` account -============================= - -In order to expose the Giskard Hub to the internet, you would need to perform the following steps - -1. Sign up `here `__ -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -You will be prompted by the following: - -.. image:: ../../assets/ngrok_aut.png - :width: 400 - -You would need to have either :code:`Google Authenticator` or :code:`1Password` on your phone to generate codes. - -2. Generate an API key `here `__ -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Copy the following key: - -.. image:: ../../assets/ngrok_aut2.png - :width: 400 - - -3. Expose the giskard hub -^^^^^^^^^^^^^^^^^^^^^^^^^ -Now you can run :code:`giskard hub expose --ngrok-token ` which should prompt you with the following instructions::: - - Exposing Giskard Hub to the internet... - Giskard Hub is now exposed to the internet. - You can now upload objects to the Giskard Hub using the following client: - - token=... - client = giskard.GiskardClient("", token) - - # To run your model with the current Python environment on Google Colab, execute these lines: - - %env GSK_API_KEY=... - !giskard worker start -d -u --name - - # To let Giskard Hub run your model in a managed Python environment, execute these lines: - - %env GSK_API_KEY=... - !giskard worker start -s -u --name - diff --git a/docs/getting_started/index.md b/docs/getting_started/index.md index e1d82c4601..85d0ad9783 100644 --- a/docs/getting_started/index.md +++ b/docs/getting_started/index.md @@ -4,17 +4,17 @@ Giskard is a **holistic Testing platform for AI models** to control all 3 types It addresses the following challenges in AI testing: -* Edge cases in AI are **domain-specific** and often seemingly **infinite**. -* The AI development process is an experimental, **trial-and-error** process where quality KPIs are multi-dimensional. -* Generative AI introduces new **security vulnerabilities** which requires constant vigilance and adversarial red-teaming. -* AI compliance with new regulations necessitate that data scientists write **extensive documentation**. +- Edge cases in AI are **domain-specific** and often seemingly **infinite**. +- The AI development process is an experimental, **trial-and-error** process where quality KPIs are multi-dimensional. +- Generative AI introduces new **security vulnerabilities** which requires constant vigilance and adversarial red-teaming. +- AI compliance with new regulations necessitate that data scientists write **extensive documentation**. Giskard provides a platform for testing all AI models, from tabular ML to LLMs. This enables AI teams to: + 1. **Reduce AI risks** by enhancing the test coverage on quality & security dimensions. 2. **Save time** by automating testing, evaluation and debugging processes. 3. **Automate compliance** with the EU AI Act and upcoming AI regulations & standards. - ## Giskard Library (open-source) An **open-source** library to scan your AI models for vulnerabilities and generate test suites automatically to aid in the Quality & Security evaluation process of ML models and LLMs. @@ -26,39 +26,16 @@ To help you solve these challenges, Giskard library helps to: - **Scan your model to find hidden vulnerabilities automatically**: The `giskard` scan automatically detects vulnerabilities such as performance bias, hallucination, prompt injection, data leakage, spurious correlation, overconfidence, etc. -

+

- - **Instantaneously generate domain-specific tests**: `giskard` automatically generates relevant, customizable tests based on the -vulnerabilities detected in the scan. + vulnerabilities detected in the scan.

- - **Integrate and automate** testing of AI models in **CI/CD** pipelines by leveraging native `giskard` integrations.

- Get started **now** with our [quickstart notebooks](../getting_started/quickstart/index.md)! ⚡️ - -## Giskard Hub - -An Enterprise Hub for teams to collaborate on top of the open-source Giskard library, with interactive testing dashboards, debugging interfaces with explainability & human feedback, and secure access controls for compliance audits. - -- 🔍 **Debug** your issues by inspecting the failing examples of your tests (⬇️ see below the DEBUG button) -

- ![](../assets/test_suite_tabular.png) - -- 📖 Leverage the Quality Assurance best practices of the most advanced ML teams with a centralized **catalog** of tests -

- ![](../assets/catalog.png) - -- 💡 Create hundreds of domain-specific tests thanks to **automated model insights** (⬇️ see below the bulbs 💡). -

- ![](../assets/push.png) - -- 💬 **Collect business feedback** and **share your model results** with data scientists, QA teams and auditors. -

- ![](../assets/credit_scoring_comment.png) diff --git a/docs/getting_started/quickstart/quickstart_llm.ipynb b/docs/getting_started/quickstart/quickstart_llm.ipynb index a092edfd7f..1668626b39 100644 --- a/docs/getting_started/quickstart/quickstart_llm.ipynb +++ b/docs/getting_started/quickstart/quickstart_llm.ipynb @@ -2928,168 +2928,6 @@ "test_suite = full_report.generate_test_suite(name=\"Test suite generated by scan\")\n", "test_suite.run()" ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false, - "id": "xrCXjxV3DlLA" - }, - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that covers a first layer of potential vulnerabilities for your LLM. From here, we encourage you to boost the coverage rate of your tests to anticipate as many failures as possible for your model. The base layer provided by the scan needs to be fine-tuned and augmented by human review, which is a great reason to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just fine-tuning tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models and prompts to decide which model or prompt to promote\n", - "* Test out input prompts and evaluation criteria that make your model fail\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false, - "id": "QiV8EoApDlLA" - }, - "source": [ - "Here's a sneak peek of the fine-tuning interface proposed by the Giskard Hub:" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "UEd30Uj_Xi3_" - }, - "source": [ - "![](../../_static/test_suite_example.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Adding persistence to our Giskard Model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To work with the Giskard Hub we need to be able to save and load the model, so that we can upload it and store. The `giskard.Model` class handles this automatically in most cases, but for more complex models you might need to do a bit of refactoring to get it working smoothly. This is especially the case if your model includes a custom index or other custom objects that are not easily serializable.\n", - "\n", - "In our case, we are using the FAISS index to retrieve the documents, and we need to tell Giskard how to save and load it. Luckily, Giskard provides a simple but powerful way to customize the model wrapper by extending the `giskard.Model` class. To make sure that we can save to disk out model, we will need to implement `save_model` and `load_model` method to save and load both the RetrievalQA and the FAISS index:" - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "metadata": {}, - "outputs": [], - "source": [ - "from pathlib import Path\n", - "from langchain.chains import load_chain\n", - "\n", - "\n", - "class FAISSRAGModel(giskard.Model):\n", - " def model_predict(self, df: pd.DataFrame):\n", - " # Same as our model_predict function above, but now using self.model,\n", - " # which we pass upon initialization.\n", - " return [self.model.run({\"query\": question}) for question in df[\"question\"]]\n", - "\n", - " def save_model(self, path: str, *args, **kwargs):\n", - " \"\"\"Saves the model object to the given directory.\"\"\"\n", - " out_dest = Path(path)\n", - "\n", - " # Save the langchain RetrievalQA object\n", - " self.model.save(out_dest.joinpath(\"model.json\"))\n", - "\n", - " # Save the FAISS-based retriever\n", - " db = self.model.retriever.vectorstore\n", - " db.save_local(out_dest.joinpath(\"faiss\"))\n", - "\n", - " @classmethod\n", - " def load_model(cls, path: str, *args, **kwargs):\n", - " \"\"\"Loads the model object from the given directory.\"\"\"\n", - " src = Path(path)\n", - "\n", - " # Load the FAISS-based retriever\n", - " db = FAISS.load_local(src.joinpath(\"faiss\"), OpenAIEmbeddings())\n", - "\n", - " # Load the chain, passing the retriever\n", - " chain = load_chain(src.joinpath(\"model.json\"), retriever=db.as_retriever())\n", - " return chain" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we can wrap our model function as above, but using our custom model class:" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "metadata": {}, - "outputs": [], - "source": [ - "giskard_model = FAISSRAGModel(\n", - " climate_qa_chain,\n", - " model_type=\"text_generation\",\n", - " name=\"Climate Change Question Answering\",\n", - " description=\"This model answers any question about climate change based on IPCC reports\",\n", - " feature_names=[\"question\"],\n", - ")\n", - "\n", - "# Let’s set this as our test suite model\n", - "test_suite.default_params[\"model\"] = giskard_model" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false, - "id": "C3EtWiq0DlLC" - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model & tests to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "l71Vs1Q1DlLC", - "outputId": "8a5708fb-7736-4e61-ed10-32c9c069e70f" - }, - "outputs": [], - "source": [ - "from giskard import GiskardClient\n", - "\n", - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" # This can be found in the Settings tab of the Giskard Hub\n", - "hf_token = \"\" # If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "my_project = client.create_project(\"my_project\", \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, \"my_project\")" - ] } ], "metadata": { diff --git a/docs/getting_started/quickstart/quickstart_nlp.ipynb b/docs/getting_started/quickstart/quickstart_nlp.ipynb index 19eda182d9..09a8d870b7 100644 --- a/docs/getting_started/quickstart/quickstart_nlp.ipynb +++ b/docs/getting_started/quickstart/quickstart_nlp.ipynb @@ -23,11 +23,7 @@ "Outline:\n", "\n", "* Detect vulnerabilities automatically with Giskard's scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to:\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -83,7 +79,7 @@ "from scipy.special import softmax\n", "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n", "\n", - "from giskard import Dataset, Model, scan, testing, GiskardClient, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { @@ -327,11 +323,11 @@ }, { "cell_type": "markdown", + "id": "9dd5baaaa6a7ee62", "metadata": {}, "source": [ "If you are running in a notebook, you can display the scan report directly in the notebook using `display(...)`, otherwise you can export the report to an HTML file. Check the [API Reference](https://docs.giskard.ai/en/stable/reference/scan/report.html#giskard.scanner.report.ScanReport) for more details on the export methods available on the `ScanReport` class." - ], - "id": "9dd5baaaa6a7ee62" + ] }, { "cell_type": "code", @@ -1851,126 +1847,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "id": "3f43ed2aac0d94a4", - "metadata": { - "collapsed": false, - "id": "3f43ed2aac0d94a4" - }, - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ] - }, - { - "cell_type": "markdown", - "id": "rm5e4aDrW_lm", - "metadata": { - "id": "rm5e4aDrW_lm" - }, - "source": [ - "Here's a sneak peak of automated model insights on a credit scoring classification model." - ] - }, - { - "cell_type": "markdown", - "id": "iZ8HF7pmWzeO", - "metadata": { - "id": "iZ8HF7pmWzeO" - }, - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ] - }, - { - "cell_type": "markdown", - "id": "iwBobi_wW25Y", - "metadata": { - "id": "iwBobi_wW25Y" - }, - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ] - }, - { - "cell_type": "markdown", - "id": "ivXicvcMVdOM", - "metadata": { - "id": "ivXicvcMVdOM" - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "829536054bde5075", - "metadata": { - "id": "829536054bde5075" - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" # This can be found in the Settings tab of the Giskard hub\n", - "# hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "my_project = client.create_project(\"my_project\", \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, \"my_project\")" - ] - }, - { - "cell_type": "markdown", - "id": "6QCJMgauXPlc", - "metadata": { - "id": "6QCJMgauXPlc" - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "uNUQSpb5XUsE", - "metadata": { - "id": "uNUQSpb5XUsE" - }, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, \"my_project\", 1753)\n", - "test_suite_downloaded.run()" - ] } ], "metadata": { diff --git a/docs/getting_started/quickstart/quickstart_tabular.ipynb b/docs/getting_started/quickstart/quickstart_tabular.ipynb index b22589b318..865914486d 100644 --- a/docs/getting_started/quickstart/quickstart_tabular.ipynb +++ b/docs/getting_started/quickstart/quickstart_tabular.ipynb @@ -20,11 +20,7 @@ "Outline:\n", "\n", "* Detect vulnerabilities automatically with Giskard's scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to:\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -38,7 +34,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "id": "774e195a6fdbaa27", "metadata": { "ExecuteTime": { @@ -64,7 +60,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "7e53bd53b81a6b37", "metadata": { "collapsed": false @@ -74,7 +70,7 @@ "import numpy as np\n", "import pandas as pd\n", "\n", - "from giskard import Model, Dataset, scan, testing, GiskardClient, demo, Suite" + "from giskard import Model, Dataset, scan, testing, demo" ] }, { @@ -89,7 +85,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "id": "54a2d07ad1ee745a", "metadata": { "ExecuteTime": { @@ -128,7 +124,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 3, "id": "1c439243e2552799", "metadata": { "ExecuteTime": { @@ -156,7 +152,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "id": "a3fea25c991fe05c", "metadata": { "ExecuteTime": { @@ -198,7 +194,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "id": "c612f436dacff7c2", "metadata": { "collapsed": false @@ -222,7 +218,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "id": "b09b62fba752975a", "metadata": { "ExecuteTime": { @@ -295,15 +291,15 @@ }, { "cell_type": "markdown", + "id": "28272f36e73f8a76", "metadata": {}, "source": [ "If you are running in a notebook, you can display the scan report directly in the notebook using `display(...)`, otherwise you can export the report to an HTML file. Check the [API Reference](https://docs.giskard.ai/en/stable/reference/scan/report.html#giskard.scanner.report.ScanReport) for more details on the export methods available on the `ScanReport` class." - ], - "id": "28272f36e73f8a76" + ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "id": "ecb49fa5", "metadata": { "ExecuteTime": { @@ -315,7 +311,7 @@ { "data": { "text/html": [ - "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -522,51 +2024,331 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", "execution_count": 14, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:25:12.590245Z", "start_time": "2023-11-08T22:24:56.084975Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Invariance to “Switch Gender”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 13:27:10,197 pid:61763 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 13:27:10,199 pid:61763 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (375, 2) executed in 0:00:00.007260\n", + "2024-05-29 13:27:10,212 pid:61763 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 13:27:10,214 pid:61763 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (375, 2) executed in 0:00:00.005033\n", + "2024-05-29 13:27:10,219 pid:61763 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:00.033015\n", + "2024-05-29 13:27:10,220 pid:61763 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.000322\n", + "Executed 'Invariance to “Switch Gender”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", " Test failed\n", " Metric: 0.95\n", - " - [TestMessageLevel.INFO] 20 rows were perturbed\n", + " - [INFO] 20 rows were perturbed\n", " \n", - "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 13:27:10,228 pid:61763 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 13:27:10,229 pid:61763 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (375, 2) executed in 0:00:00.004654\n", + "2024-05-29 13:27:10,248 pid:61763 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 13:27:13,896 pid:61763 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (375, 2) executed in 0:00:03.650915\n", + "2024-05-29 13:27:13,899 pid:61763 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:03.676607\n", + "2024-05-29 13:27:13,900 pid:61763 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.000276\n", + "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", " Test failed\n", - " Metric: 0.87\n", - " - [TestMessageLevel.INFO] 352 rows were perturbed\n", + " Metric: 0.85\n", + " - [INFO] 355 rows were perturbed\n", " \n", - "Executed 'Overconfidence on data slice “`avg_whitespace(text)` < 0.149”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7140350877192984, 'p_threshold': 0.43497172683775553}: \n", + "2024-05-29 13:27:13,909 pid:61763 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 13:27:13,910 pid:61763 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (87, 2) executed in 0:00:00.006215\n", + "Executed 'Overconfidence on data slice “`text_length(text)` < 65.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6886956521739132, 'p_threshold': 0.43497172683775553}: \n", " Test failed\n", - " Metric: 0.76\n", + " Metric: 0.74\n", " \n", - " \n" + " \n", + "2024-05-29 13:27:13,913 pid:61763 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 13:27:13,913 pid:61763 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 13:27:13,914 pid:61763 MainThread giskard.core.suite INFO Invariance to “Switch Gender” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.95}\n", + "2024-05-29 13:27:13,914 pid:61763 MainThread giskard.core.suite INFO Invariance to “Add typos” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.8450704225352113}\n", + "2024-05-29 13:27:13,914 pid:61763 MainThread giskard.core.suite INFO Overconfidence on data slice “`text_length(text)` < 65.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6886956521739132, 'p_threshold': 0.43497172683775553}): {failed, metric=0.7419354838709677}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Invariance to “Switch Gender”\n
\n \n Measured Metric = 0.95\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n e22c8893-a3d6-4ecd-8341-7cab2b8c311d\n
\n \n
\n dataset\n Tweets sentiment dataset\n
\n \n
\n transformation_function\n Switch Gender\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Invariance to “Add typos”\n
\n \n Measured Metric = 0.87216\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n e22c8893-a3d6-4ecd-8341-7cab2b8c311d\n
\n \n
\n dataset\n Tweets sentiment dataset\n
\n \n
\n transformation_function\n Add typos\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`avg_whitespace(text)` < 0.149”\n
\n \n Measured Metric = 0.7561\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n e22c8893-a3d6-4ecd-8341-7cab2b8c311d\n
\n \n
\n dataset\n Tweets sentiment dataset\n
\n \n
\n slicing_function\n `avg_whitespace(text)` < 0.149\n
\n \n
\n threshold\n 0.7140350877192984\n
\n \n
\n p_threshold\n 0.43497172683775553\n
\n \n
\n
\n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Switch Gender”\n", + "
\n", + " \n", + " Measured Metric = 0.95\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Twitter sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Tweets sentiment dataset\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Switch Gender\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Add typos”\n", + "
\n", + " \n", + " Measured Metric = 0.84507\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Twitter sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Tweets sentiment dataset\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Add typos\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`text_length(text)` < 65.500”\n", + "
\n", + " \n", + " Measured Metric = 0.74194\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Twitter sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Tweets sentiment dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text_length(text)` < 65.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6886956521739132\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.43497172683775553\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 14, "metadata": {}, @@ -607,118 +2389,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ] - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { diff --git a/docs/reference/notebooks/amazon_review_classification_sklearn.ipynb b/docs/reference/notebooks/amazon_review_classification_sklearn.ipynb index 5d4d002cdf..836b41bf7b 100644 --- a/docs/reference/notebooks/amazon_review_classification_sklearn.ipynb +++ b/docs/reference/notebooks/amazon_review_classification_sklearn.ipynb @@ -21,12 +21,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -41,7 +36,7 @@ }, { "cell_type": "code", - "execution_count": 40, + "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2023-08-21T10:23:41.394690Z", @@ -63,13 +58,13 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 1, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T20:55:52.227278Z", "start_time": "2023-11-08T20:55:52.178313Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -86,33 +81,33 @@ "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import FunctionTransformer\n", "\n", - "from giskard import Dataset, Model, GiskardClient, testing, Suite, scan" + "from giskard import Dataset, Model, scan, testing" ] }, { "cell_type": "markdown", - "source": [ - "## Notebook-level settings" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Notebook-level settings" + ] }, { "cell_type": "code", - "execution_count": 3, - "outputs": [], - "source": [ - "# Disable chained assignment warning.\n", - "pd.options.mode.chained_assignment = None" - ], + "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T20:55:54.918094Z", "start_time": "2023-11-08T20:55:54.886389Z" - } - } + }, + "collapsed": false + }, + "outputs": [], + "source": [ + "# Disable chained assignment warning.\n", + "pd.options.mode.chained_assignment = None" + ] }, { "cell_type": "markdown", @@ -123,7 +118,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2023-11-08T20:55:58.032511Z", @@ -162,13 +157,13 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 4, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T20:56:00.293536Z", "start_time": "2023-11-08T20:56:00.234306Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -240,7 +235,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2023-11-08T20:56:57.696227Z", @@ -264,7 +259,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2023-11-08T20:57:20.112511Z", @@ -300,7 +295,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2023-11-08T20:57:21.472626Z", @@ -368,7 +363,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Wrap prediction function so that the whole pipeline is uploaded to the Hub\n", + "# Wrap prediction function\n", "def prediction_function(df):\n", " return pipeline.predict_proba(df)\n", "\n", @@ -391,12 +386,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -418,18 +413,2369 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 12, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T21:19:07.555323Z", "start_time": "2023-11-08T21:19:07.343077Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -441,12 +2787,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Generate comprehensive test suites automatically for your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Generate comprehensive test suites automatically for your model" + ] }, { "cell_type": "markdown", @@ -461,72 +2807,593 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 13, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T21:31:48.577324Z", "start_time": "2023-11-08T21:31:39.914226Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 11:31:46,497 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:46,501 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (9587, 2) executed in 0:00:00.015202\n", + "2024-05-29 11:31:47,278 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,490 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (9587, 2) executed in 0:00:00.246625\n", + "2024-05-29 11:31:47,499 pid:47376 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:01.872730\n", + "2024-05-29 11:31:47,500 pid:47376 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.000723\n", + "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", " Test failed\n", - " Metric: 0.91\n", - " - [TestMessageLevel.INFO] 9506 rows were perturbed\n", + " Metric: 0.9\n", + " - [INFO] 9515 rows were perturbed\n", " \n", - "Executed 'Overconfidence on data slice “`reviewText` contains \"don\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.20729267505646984, 'p_threshold': 0.5}: \n", + "2024-05-29 11:31:47,559 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,560 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1554, 2) executed in 0:00:00.013542\n", + "Executed 'Overconfidence on data slice “`reviewText` contains \"don\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.20769479469770452, 'p_threshold': 0.5}: \n", " Test failed\n", " Metric: 0.27\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`text_length(reviewText)` < 174.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.20729267505646984, 'p_threshold': 0.5}: \n", + "2024-05-29 11:31:47,582 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,584 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (4436, 2) executed in 0:00:00.016746\n", + "Executed 'Overconfidence on data slice “`text_length(reviewText)` < 174.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.20769479469770452, 'p_threshold': 0.5}: \n", " Test failed\n", " Metric: 0.21\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`reviewText` contains \"better\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03855220611244394, 'p_threshold': 0.95}: \n", - " Test failed\n", - " Metric: 0.05\n", - " \n", - " \n", - "Executed 'Underconfidence on data slice “`reviewText` contains \"got\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03855220611244394, 'p_threshold': 0.95}: \n", + "2024-05-29 11:31:47,641 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,642 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (734, 2) executed in 0:00:00.008658\n", + "Executed 'Underconfidence on data slice “`reviewText` contains \"way\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.05\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`reviewText` contains \"doesn\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03855220611244394, 'p_threshold': 0.95}: \n", + "2024-05-29 11:31:47,698 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,699 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (707, 2) executed in 0:00:00.008718\n", + "Executed 'Underconfidence on data slice “`reviewText` contains \"better\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.04\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`reviewText` contains \"way\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03855220611244394, 'p_threshold': 0.95}: \n", + "2024-05-29 11:31:47,756 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,757 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (849, 2) executed in 0:00:00.009271\n", + "Executed 'Underconfidence on data slice “`reviewText` contains \"want\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.04\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`reviewText` contains \"good\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03855220611244394, 'p_threshold': 0.95}: \n", + "2024-05-29 11:31:47,813 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,814 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (733, 2) executed in 0:00:00.007101\n", + "Executed 'Underconfidence on data slice “`reviewText` contains \"got\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.04\n", " \n", " \n", - "Executed 'Recall on data slice “`reviewText` contains \"download\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6416057519472739}: \n", + "2024-05-29 11:31:47,868 pid:47376 MainThread giskard.datasets.base INFO Casting dataframe columns from {'reviewText': 'object'} to {'reviewText': 'object'}\n", + "2024-05-29 11:31:47,869 pid:47376 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (538, 2) executed in 0:00:00.006875\n", + "Executed 'Recall on data slice “`reviewText` contains \"download\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6419472738166567}: \n", " Test failed\n", " Metric: 0.59\n", " \n", - " \n" + " \n", + "2024-05-29 11:31:47,872 pid:47376 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 11:31:47,873 pid:47376 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 11:31:47,873 pid:47376 MainThread giskard.core.suite INFO Invariance to “Add typos” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.9038360483447189}\n", + "2024-05-29 11:31:47,873 pid:47376 MainThread giskard.core.suite INFO Overconfidence on data slice “`reviewText` contains \"don\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.20769479469770452, 'p_threshold': 0.5}): {failed, metric=0.26622296173044924}\n", + "2024-05-29 11:31:47,873 pid:47376 MainThread giskard.core.suite INFO Overconfidence on data slice “`text_length(reviewText)` < 174.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.20769479469770452, 'p_threshold': 0.5}): {failed, metric=0.21353196772191185}\n", + "2024-05-29 11:31:47,874 pid:47376 MainThread giskard.core.suite INFO Underconfidence on data slice “`reviewText` contains \"way\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}): {failed, metric=0.04904632152588556}\n", + "2024-05-29 11:31:47,874 pid:47376 MainThread giskard.core.suite INFO Underconfidence on data slice “`reviewText` contains \"better\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}): {failed, metric=0.042432814710042434}\n", + "2024-05-29 11:31:47,874 pid:47376 MainThread giskard.core.suite INFO Underconfidence on data slice “`reviewText` contains \"want\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}): {failed, metric=0.04240282685512368}\n", + "2024-05-29 11:31:47,874 pid:47376 MainThread giskard.core.suite INFO Underconfidence on data slice “`reviewText` contains \"got\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.03912589965578388, 'p_threshold': 0.95}): {failed, metric=0.03956343792633015}\n", + "2024-05-29 11:31:47,875 pid:47376 MainThread giskard.core.suite INFO Recall on data slice “`reviewText` contains \"download\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6419472738166567}): {failed, metric=0.5929203539823009}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Invariance to “Add typos”\n
\n \n Measured Metric = 0.90627\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n transformation_function\n Add typos\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`reviewText` contains "don"”\n
\n \n Measured Metric = 0.26578\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `reviewText` contains "don"\n
\n \n
\n threshold\n 0.20729267505646984\n
\n \n
\n p_threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`text_length(reviewText)` < 174.500”\n
\n \n Measured Metric = 0.2119\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `text_length(reviewText)` < 174.500\n
\n \n
\n threshold\n 0.20729267505646984\n
\n \n
\n p_threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Underconfidence on data slice “`reviewText` contains "better"”\n
\n \n Measured Metric = 0.04526\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `reviewText` contains "better"\n
\n \n
\n threshold\n 0.03855220611244394\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Underconfidence on data slice “`reviewText` contains "got"”\n
\n \n Measured Metric = 0.04502\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `reviewText` contains "got"\n
\n \n
\n threshold\n 0.03855220611244394\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Underconfidence on data slice “`reviewText` contains "doesn"”\n
\n \n Measured Metric = 0.0415\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `reviewText` contains "doesn"\n
\n \n
\n threshold\n 0.03855220611244394\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Underconfidence on data slice “`reviewText` contains "way"”\n
\n \n Measured Metric = 0.04087\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `reviewText` contains "way"\n
\n \n
\n threshold\n 0.03855220611244394\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Underconfidence on data slice “`reviewText` contains "good"”\n
\n \n Measured Metric = 0.03938\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `reviewText` contains "good"\n
\n \n
\n threshold\n 0.03855220611244394\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`reviewText` contains "download"”\n
\n \n Measured Metric = 0.59292\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 2bbb4689-1208-4860-971f-add01cc7f3eb\n
\n \n
\n dataset\n reviews\n
\n \n
\n slicing_function\n `reviewText` contains "download"\n
\n \n
\n threshold\n 0.6416057519472739\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Add typos”\n", + "
\n", + " \n", + " Measured Metric = 0.90384\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Add typos\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`reviewText` contains "don"”\n", + "
\n", + " \n", + " Measured Metric = 0.26622\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `reviewText` contains "don"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.20769479469770452\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`text_length(reviewText)` < 174.500”\n", + "
\n", + " \n", + " Measured Metric = 0.21353\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text_length(reviewText)` < 174.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.20769479469770452\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`reviewText` contains "way"”\n", + "
\n", + " \n", + " Measured Metric = 0.04905\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `reviewText` contains "way"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.03912589965578388\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`reviewText` contains "better"”\n", + "
\n", + " \n", + " Measured Metric = 0.04243\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `reviewText` contains "better"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.03912589965578388\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`reviewText` contains "want"”\n", + "
\n", + " \n", + " Measured Metric = 0.0424\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `reviewText` contains "want"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.03912589965578388\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`reviewText` contains "got"”\n", + "
\n", + " \n", + " Measured Metric = 0.03956\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `reviewText` contains "got"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.03912589965578388\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`reviewText` contains "download"”\n", + "
\n", + " \n", + " Measured Metric = 0.59292\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " review_helpfulness_predictor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " reviews\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `reviewText` contains "download"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6419472738166567\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 16, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } @@ -565,113 +3432,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to: \n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ] - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -690,7 +3450,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.11" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/cancer_detection_xgboost.ipynb b/docs/reference/notebooks/cancer_detection_xgboost.ipynb index 00c454ef4b..b697c85755 100644 --- a/docs/reference/notebooks/cancer_detection_xgboost.ipynb +++ b/docs/reference/notebooks/cancer_detection_xgboost.ipynb @@ -2,6 +2,10 @@ "cells": [ { "cell_type": "markdown", + "id": "990eccb8", + "metadata": { + "collapsed": false + }, "source": [ "# Breast cancer detection [XGBoost]\n", "\n", @@ -18,17 +22,8 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" - ], - "metadata": { - "collapsed": false - }, - "id": "990eccb8" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" + ] }, { "attachments": {}, @@ -42,7 +37,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": null, "id": "eb828d6da954f51d", "metadata": { "ExecuteTime": { @@ -83,37 +78,37 @@ "from sklearn.model_selection import train_test_split\n", "from xgboost import XGBClassifier\n", "\n", - "from giskard import Dataset, Model, scan, testing, Suite, GiskardClient" + "from giskard import Dataset, Model, scan, testing" ] }, { "cell_type": "markdown", - "source": [ - "## Define constants" - ], + "id": "9dac8b68bec87e9d", "metadata": { "collapsed": false }, - "id": "9dac8b68bec87e9d" + "source": [ + "## Define constants" + ] }, { "cell_type": "code", "execution_count": 2, + "id": "6f78eeb5c5d1e734", + "metadata": { + "ExecuteTime": { + "end_time": "2023-11-08T22:42:48.116285Z", + "start_time": "2023-11-08T22:42:48.066331Z" + }, + "collapsed": false + }, "outputs": [], "source": [ "# Constants.\n", "TARGET_COLUMN_NAME = \"target\"\n", "\n", "RANDOM_SEED = 42" - ], - "metadata": { - "collapsed": false, - "ExecuteTime": { - "end_time": "2023-11-08T22:42:48.116285Z", - "start_time": "2023-11-08T22:42:48.066331Z" - } - }, - "id": "6f78eeb5c5d1e734" + ] }, { "attachments": {}, @@ -126,13 +121,13 @@ }, { "cell_type": "markdown", - "source": [ - "### Load data" - ], + "id": "7a271dcc766e688f", "metadata": { "collapsed": false }, - "id": "7a271dcc766e688f" + "source": [ + "### Load data" + ] }, { "cell_type": "code", @@ -153,46 +148,54 @@ }, { "cell_type": "markdown", - "source": [ - "### Train-test split" - ], + "id": "bb43fe6e0c8de713", "metadata": { "collapsed": false }, - "id": "bb43fe6e0c8de713" + "source": [ + "### Train-test split" + ] }, { "cell_type": "code", "execution_count": 4, - "outputs": [], - "source": [ - "# Train/test split.\n", - "X_train, X_test, y_train, y_test = train_test_split(features.loc[:, features.columns != TARGET_COLUMN_NAME], target,\n", - " random_state=RANDOM_SEED)" - ], + "id": "b3cd0e34e8eb0a25", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:42:50.702481Z", "start_time": "2023-11-08T22:42:50.618451Z" - } + }, + "collapsed": false }, - "id": "b3cd0e34e8eb0a25" + "outputs": [], + "source": [ + "# Train/test split.\n", + "X_train, X_test, y_train, y_test = train_test_split(features.loc[:, features.columns != TARGET_COLUMN_NAME], target,\n", + " random_state=RANDOM_SEED)" + ] }, { "cell_type": "markdown", - "source": [ - "### Wrap dataset with Giskard\n", - "To prepare for the vulnerability scan, make sure to wrap your dataset using Giskard's Dataset class. More details [here](https://docs.giskard.ai/en/stable/open_source/scan/scan_nlp/index.html#step-1-wrap-your-dataset)." - ], + "id": "981e2b28aa4fda32", "metadata": { "collapsed": false }, - "id": "981e2b28aa4fda32" + "source": [ + "### Wrap dataset with Giskard\n", + "To prepare for the vulnerability scan, make sure to wrap your dataset using Giskard's Dataset class. More details [here](https://docs.giskard.ai/en/stable/open_source/scan/scan_nlp/index.html#step-1-wrap-your-dataset)." + ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, + "id": "e3c3e6a5", + "metadata": { + "ExecuteTime": { + "end_time": "2023-11-08T22:42:51.594977Z", + "start_time": "2023-11-08T22:42:51.547696Z" + }, + "collapsed": false + }, "outputs": [], "source": [ "raw_data = pd.concat([X_test, y_test], axis=1)\n", @@ -202,15 +205,7 @@ " target=\"target\", # Ground truth variable.\n", " name=\"breast_cancer\", # Optional.\n", ")" - ], - "metadata": { - "collapsed": false, - "ExecuteTime": { - "end_time": "2023-11-08T22:42:51.594977Z", - "start_time": "2023-11-08T22:42:51.547696Z" - } - }, - "id": "e3c3e6a5" + ] }, { "attachments": {}, @@ -278,13 +273,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], + "id": "b4fd45501d666893", "metadata": { "collapsed": false }, - "id": "b4fd45501d666893" + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -321,7 +316,3439 @@ "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -351,7 +3778,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, "id": "bea736a9", "metadata": { "ExecuteTime": { @@ -364,136 +3791,1376 @@ "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Nominal association (Theil's U) on data slice “`worst concave points` >= 0.144”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,362 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,366 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.007008\n", + "Executed 'Nominal association (Theil's U) on data slice “`worst concave points` >= 0.144”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.77\n", - " - [TestMessageLevel.INFO] metric = 0.7745014140794255, threshold = 0.5\n", + " - [INFO] metric = 0.7745014140794255, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`worst radius` >= 17.420”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,374 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,376 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.005423\n", + "Executed 'Nominal association (Theil's U) on data slice “`worst radius` >= 17.420”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.76\n", - " - [TestMessageLevel.INFO] metric = 0.7639160128151123, threshold = 0.5\n", + " - [INFO] metric = 0.7639160128151123, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`worst perimeter` >= 110.950”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,385 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,388 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.005598\n", + "Executed 'Nominal association (Theil's U) on data slice “`worst perimeter` >= 110.950”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.74\n", - " - [TestMessageLevel.INFO] metric = 0.7427987555404146, threshold = 0.5\n", + " - [INFO] metric = 0.7427987555404146, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`worst area` >= 885.950”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,394 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,399 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.006882\n", + "Executed 'Nominal association (Theil's U) on data slice “`worst area` >= 885.950”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.71\n", - " - [TestMessageLevel.INFO] metric = 0.7131111241996619, threshold = 0.5\n", + " - [INFO] metric = 0.7131111241996619, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`mean concave points` >= 0.053”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,406 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,409 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.005363\n", + "Executed 'Nominal association (Theil's U) on data slice “`mean concave points` >= 0.053”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.64\n", - " - [TestMessageLevel.INFO] metric = 0.6377395426892722, threshold = 0.5\n", + " - [INFO] metric = 0.6377395426892722, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`mean perimeter` >= 96.380”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,419 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,424 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.009371\n", + "Executed 'Nominal association (Theil's U) on data slice “`mean perimeter` >= 96.380”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.62\n", - " - [TestMessageLevel.INFO] metric = 0.6227293846650519, threshold = 0.5\n", + " - [INFO] metric = 0.6227293846650519, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`mean radius` >= 15.060”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,433 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,439 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.009926\n", + "Executed 'Nominal association (Theil's U) on data slice “`mean area` >= 697.300”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.61\n", - " - [TestMessageLevel.INFO] metric = 0.6087180717056299, threshold = 0.5\n", + " - [INFO] metric = 0.6087180717056299, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`mean area` >= 697.300”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,449 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,455 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.010707\n", + "Executed 'Nominal association (Theil's U) on data slice “`mean radius` >= 15.060”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.61\n", - " - [TestMessageLevel.INFO] metric = 0.6087180717056299, threshold = 0.5\n", + " - [INFO] metric = 0.6087180717056299, threshold = 0.5\n", " \n", - "Executed 'Nominal association (Theil's U) on data slice “`area error` >= 39.690”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", + "2024-05-29 11:38:26,463 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,467 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (143, 31) executed in 0:00:00.007197\n", + "Executed 'Nominal association (Theil's U) on data slice “`area error` >= 39.690”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}: \n", " Test failed\n", " Metric: 0.52\n", - " - [TestMessageLevel.INFO] metric = 0.5166070702259753, threshold = 0.5\n", + " - [INFO] metric = 0.5166070702259753, threshold = 0.5\n", " \n", - "Executed 'Accuracy on data slice “`worst radius` >= 14.765 AND `worst radius` < 17.625”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,475 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,481 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.009332\n", + "Executed 'Accuracy on data slice “`worst radius` >= 14.765 AND `worst radius` < 17.625”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Accuracy on data slice “`mean radius` >= 13.310 AND `mean radius` < 15.005”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,493 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,497 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.008997\n", + "Executed 'Accuracy on data slice “`worst perimeter` >= 96.625 AND `worst perimeter` < 122.350”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Accuracy on data slice “`mean perimeter` >= 86.140 AND `mean perimeter` < 98.085”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,507 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,511 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.008990\n", + "Executed 'Accuracy on data slice “`mean perimeter` >= 86.140 AND `mean perimeter` < 98.085”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Accuracy on data slice “`worst perimeter` >= 96.625 AND `worst perimeter` < 122.350”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,523 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,527 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.008275\n", + "Executed 'Accuracy on data slice “`mean area` >= 518.300 AND `mean area` < 664.350”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Accuracy on data slice “`mean area` >= 518.300 AND `mean area` < 664.350”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,538 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,541 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.010876\n", + "Executed 'Accuracy on data slice “`mean radius` >= 13.310 AND `mean radius` < 15.005”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Accuracy on data slice “`mean concavity` >= 0.078 AND `mean concavity` < 0.140”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,562 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,577 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.020399\n", + "Executed 'Accuracy on data slice “`mean concave points` >= 0.048 AND `mean concave points` < 0.079”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Accuracy on data slice “`mean concave points` >= 0.048 AND `mean concave points` < 0.079”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,592 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,601 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.013636\n", + "Executed 'Accuracy on data slice “`mean concavity` >= 0.078 AND `mean concavity` < 0.140”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Accuracy on data slice “`worst area` >= 610.200 AND `worst area` < 828.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,616 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,625 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (33, 31) executed in 0:00:00.013964\n", + "Executed 'Accuracy on data slice “`worst area` >= 610.200 AND `worst area` < 828.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.82\n", " \n", " \n", - "Executed 'Accuracy on data slice “`fractal dimension error` < 0.003 AND `fractal dimension error` >= 0.002”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,637 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,649 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.019259\n", + "Executed 'Accuracy on data slice “`fractal dimension error` < 0.003 AND `fractal dimension error` >= 0.002”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.83\n", " \n", " \n", - "Executed 'Accuracy on data slice “`worst concavity` >= 0.277 AND `worst concavity` < 0.415”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,662 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,667 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 31) executed in 0:00:00.010570\n", + "Executed 'Accuracy on data slice “`worst concavity` >= 0.277 AND `worst concavity` < 0.415”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.83\n", " \n", " \n", - "Executed 'Accuracy on data slice “`compactness error` >= 0.020 AND `compactness error` < 0.030”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,667 pid:50687 Thread-54 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,668 pid:50687 Thread-47 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,674 pid:50687 Thread-48 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,675 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,681 pid:50687 Thread-50 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,684 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (32, 31) executed in 0:00:00.013371\n", + "Executed 'Accuracy on data slice “`compactness error` >= 0.020 AND `compactness error` < 0.030”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.84\n", " \n", " \n", - "Executed 'Accuracy on data slice “`mean compactness` >= 0.099 AND `mean compactness` < 0.128”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,693 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,697 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (32, 31) executed in 0:00:00.007443\n", + "Executed 'Accuracy on data slice “`mean compactness` >= 0.099 AND `mean compactness` < 0.128”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.84\n", " \n", " \n", - "Executed 'Recall on data slice “`concave points error` >= 0.009 AND `concave points error` < 0.015”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9179775280898876}: \n", + "2024-05-29 11:38:26,707 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,710 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (51, 31) executed in 0:00:00.007322\n", + "Executed 'Recall on data slice “`concave points error` >= 0.009 AND `concave points error` < 0.015”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9179775280898876}: \n", " Test failed\n", " Metric: 0.88\n", " \n", " \n", - "Executed 'Accuracy on data slice “`perimeter error` < 2.151 AND `perimeter error` >= 1.505”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,720 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,723 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (35, 31) executed in 0:00:00.008618\n", + "Executed 'Accuracy on data slice “`perimeter error` < 2.151 AND `perimeter error` >= 1.505”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.89\n", " \n", " \n", - "Executed 'Accuracy on data slice “`area error` < 40.745 AND `area error` >= 19.215”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", + "2024-05-29 11:38:26,732 pid:50687 MainThread giskard.datasets.base INFO Casting dataframe columns from {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'} to {'mean radius': 'float64', 'mean texture': 'float64', 'mean perimeter': 'float64', 'mean area': 'float64', 'mean smoothness': 'float64', 'mean compactness': 'float64', 'mean concavity': 'float64', 'mean concave points': 'float64', 'mean symmetry': 'float64', 'mean fractal dimension': 'float64', 'radius error': 'float64', 'texture error': 'float64', 'perimeter error': 'float64', 'area error': 'float64', 'smoothness error': 'float64', 'compactness error': 'float64', 'concavity error': 'float64', 'concave points error': 'float64', 'symmetry error': 'float64', 'fractal dimension error': 'float64', 'worst radius': 'float64', 'worst texture': 'float64', 'worst perimeter': 'float64', 'worst area': 'float64', 'worst smoothness': 'float64', 'worst compactness': 'float64', 'worst concavity': 'float64', 'worst concave points': 'float64', 'worst symmetry': 'float64', 'worst fractal dimension': 'float64'}\n", + "2024-05-29 11:38:26,736 pid:50687 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (45, 31) executed in 0:00:00.009279\n", + "Executed 'Accuracy on data slice “`area error` < 40.745 AND `area error` >= 19.215”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}: \n", " Test failed\n", " Metric: 0.89\n", " \n", - " \n" + " \n", + "2024-05-29 11:38:26,737 pid:50687 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 11:38:26,738 pid:50687 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 11:38:26,738 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`worst concave points` >= 0.144” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.7745014140794255}\n", + "2024-05-29 11:38:26,738 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`worst radius` >= 17.420” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.7639160128151123}\n", + "2024-05-29 11:38:26,739 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`worst perimeter` >= 110.950” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.7427987555404146}\n", + "2024-05-29 11:38:26,740 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`worst area` >= 885.950” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.7131111241996619}\n", + "2024-05-29 11:38:26,740 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`mean concave points` >= 0.053” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.6377395426892722}\n", + "2024-05-29 11:38:26,741 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`mean perimeter` >= 96.380” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.6227293846650519}\n", + "2024-05-29 11:38:26,741 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`mean area` >= 697.300” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.6087180717056299}\n", + "2024-05-29 11:38:26,743 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`mean radius` >= 15.060” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.6087180717056299}\n", + "2024-05-29 11:38:26,744 pid:50687 MainThread giskard.core.suite INFO Nominal association (Theil's U) on data slice “`area error` >= 39.690” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5}): {failed, metric=0.5166070702259753}\n", + "2024-05-29 11:38:26,745 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`worst radius` >= 14.765 AND `worst radius` < 17.625” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8}\n", + "2024-05-29 11:38:26,746 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`worst perimeter` >= 96.625 AND `worst perimeter` < 122.350” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8}\n", + "2024-05-29 11:38:26,747 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`mean perimeter` >= 86.140 AND `mean perimeter` < 98.085” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8}\n", + "2024-05-29 11:38:26,748 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`mean area` >= 518.300 AND `mean area` < 664.350” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8}\n", + "2024-05-29 11:38:26,748 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`mean radius` >= 13.310 AND `mean radius` < 15.005” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8}\n", + "2024-05-29 11:38:26,749 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`mean concave points` >= 0.048 AND `mean concave points` < 0.079” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8}\n", + "2024-05-29 11:38:26,749 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`mean concavity` >= 0.078 AND `mean concavity` < 0.140” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8}\n", + "2024-05-29 11:38:26,752 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`worst area` >= 610.200 AND `worst area` < 828.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8181818181818182}\n", + "2024-05-29 11:38:26,752 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`fractal dimension error` < 0.003 AND `fractal dimension error` >= 0.002” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8333333333333334}\n", + "2024-05-29 11:38:26,752 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`worst concavity` >= 0.277 AND `worst concavity` < 0.415” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8333333333333334}\n", + "2024-05-29 11:38:26,753 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`compactness error` >= 0.020 AND `compactness error` < 0.030” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.84375}\n", + "2024-05-29 11:38:26,753 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`mean compactness` >= 0.099 AND `mean compactness` < 0.128” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.84375}\n", + "2024-05-29 11:38:26,753 pid:50687 MainThread giskard.core.suite INFO Recall on data slice “`concave points error` >= 0.009 AND `concave points error` < 0.015” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9179775280898876}): {failed, metric=0.875}\n", + "2024-05-29 11:38:26,754 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`perimeter error` < 2.151 AND `perimeter error` >= 1.505” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8857142857142857}\n", + "2024-05-29 11:38:26,754 pid:50687 MainThread giskard.core.suite INFO Accuracy on data slice “`area error` < 40.745 AND `area error` >= 19.215” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9101398601398601}): {failed, metric=0.8888888888888888}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Nominal association (Theil's U) on data slice “`worst concave points` >= 0.144”\n
\n \n Measured Metric = 0.7745\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst concave points` >= 0.144\n
\n \n
\n threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Nominal association (Theil's U) on data slice “`worst radius` >= 17.420”\n
\n \n Measured Metric = 0.76392\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst radius` >= 17.420\n
\n \n
\n threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Nominal association (Theil's U) on data slice “`worst perimeter` >= 110.950”\n
\n \n Measured Metric = 0.7428\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst perimeter` >= 110.950\n
\n \n
\n threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Nominal association (Theil's U) on data slice “`worst area` >= 885.950”\n
\n \n Measured Metric = 0.71311\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst area` >= 885.950\n
\n \n
\n threshold\n 0.5\n
\n \n
\n \n \n \n
\n Test Nominal association (Theil's U) on data slice “`mean concave points` >= 0.053”\n
\n \n Measured Metric = 0.63774\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean concave points` >= 0.053\n
\n \n
\n threshold\n 0.5\n
\n \n
\n \n \n \n
\n Test Nominal association (Theil's U) on data slice “`mean perimeter` >= 96.380”\n
\n \n Measured Metric = 0.62273\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean perimeter` >= 96.380\n
\n \n
\n threshold\n 0.5\n
\n \n
\n \n \n \n
\n Test Nominal association (Theil's U) on data slice “`mean radius` >= 15.060”\n
\n \n Measured Metric = 0.60872\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean radius` >= 15.060\n
\n \n
\n threshold\n 0.5\n
\n \n
\n \n \n \n
\n Test Nominal association (Theil's U) on data slice “`mean area` >= 697.300”\n
\n \n Measured Metric = 0.60872\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean area` >= 697.300\n
\n \n
\n threshold\n 0.5\n
\n \n
\n \n \n \n
\n Test Nominal association (Theil's U) on data slice “`area error` >= 39.690”\n
\n \n Measured Metric = 0.51661\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `area error` >= 39.690\n
\n \n
\n threshold\n 0.5\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`worst radius` >= 14.765 AND `worst radius` < 17.625”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst radius` >= 14.765 AND `worst radius` < 17.625\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`mean radius` >= 13.310 AND `mean radius` < 15.005”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean radius` >= 13.310 AND `mean radius` < 15.005\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`mean perimeter` >= 86.140 AND `mean perimeter` < 98.085”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean perimeter` >= 86.140 AND `mean perimeter` < 98.085\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`worst perimeter` >= 96.625 AND `worst perimeter` < 122.350”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst perimeter` >= 96.625 AND `worst perimeter` < 122.350\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`mean area` >= 518.300 AND `mean area` < 664.350”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean area` >= 518.300 AND `mean area` < 664.350\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`mean concavity` >= 0.078 AND `mean concavity` < 0.140”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean concavity` >= 0.078 AND `mean concavity` < 0.140\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`mean concave points` >= 0.048 AND `mean concave points` < 0.079”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean concave points` >= 0.048 AND `mean concave points` < 0.079\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`worst area` >= 610.200 AND `worst area` < 828.500”\n
\n \n Measured Metric = 0.81818\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst area` >= 610.200 AND `worst area` < 828.500\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`fractal dimension error` < 0.003 AND `fractal dimension error` >= 0.002”\n
\n \n Measured Metric = 0.83333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `fractal dimension error` < 0.003 AND `fractal dimension error` >= 0.002\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`worst concavity` >= 0.277 AND `worst concavity` < 0.415”\n
\n \n Measured Metric = 0.83333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `worst concavity` >= 0.277 AND `worst concavity` < 0.415\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`compactness error` >= 0.020 AND `compactness error` < 0.030”\n
\n \n Measured Metric = 0.84375\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `compactness error` >= 0.020 AND `compactness error` < 0.030\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`mean compactness` >= 0.099 AND `mean compactness` < 0.128”\n
\n \n Measured Metric = 0.84375\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `mean compactness` >= 0.099 AND `mean compactness` < 0.128\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`concave points error` >= 0.009 AND `concave points error` < 0.015”\n
\n \n Measured Metric = 0.875\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `concave points error` >= 0.009 AND `concave points error` < 0.015\n
\n \n
\n threshold\n 0.9179775280898876\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`perimeter error` < 2.151 AND `perimeter error` >= 1.505”\n
\n \n Measured Metric = 0.88571\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `perimeter error` < 2.151 AND `perimeter error` >= 1.505\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`area error` < 40.745 AND `area error` >= 19.215”\n
\n \n Measured Metric = 0.88889\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 534fd047-4921-414f-9e34-ca2a918b6822\n
\n \n
\n dataset\n breast_cancer\n
\n \n
\n slicing_function\n `area error` < 40.745 AND `area error` >= 19.215\n
\n \n
\n threshold\n 0.9101398601398601\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`worst concave points` >= 0.144”\n", + "
\n", + " \n", + " Measured Metric = 0.7745\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst concave points` >= 0.144\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`worst radius` >= 17.420”\n", + "
\n", + " \n", + " Measured Metric = 0.76392\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst radius` >= 17.420\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`worst perimeter` >= 110.950”\n", + "
\n", + " \n", + " Measured Metric = 0.7428\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst perimeter` >= 110.950\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`worst area` >= 885.950”\n", + "
\n", + " \n", + " Measured Metric = 0.71311\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst area` >= 885.950\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`mean concave points` >= 0.053”\n", + "
\n", + " \n", + " Measured Metric = 0.63774\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean concave points` >= 0.053\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`mean perimeter` >= 96.380”\n", + "
\n", + " \n", + " Measured Metric = 0.62273\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean perimeter` >= 96.380\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`mean area` >= 697.300”\n", + "
\n", + " \n", + " Measured Metric = 0.60872\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean area` >= 697.300\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`mean radius` >= 15.060”\n", + "
\n", + " \n", + " Measured Metric = 0.60872\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean radius` >= 15.060\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Nominal association (Theil's U) on data slice “`area error` >= 39.690”\n", + "
\n", + " \n", + " Measured Metric = 0.51661\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `area error` >= 39.690\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`worst radius` >= 14.765 AND `worst radius` < 17.625”\n", + "
\n", + " \n", + " Measured Metric = 0.8\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst radius` >= 14.765 AND `worst radius` < 17.625\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`worst perimeter` >= 96.625 AND `worst perimeter` < 122.350”\n", + "
\n", + " \n", + " Measured Metric = 0.8\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst perimeter` >= 96.625 AND `worst perimeter` < 122.350\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`mean perimeter` >= 86.140 AND `mean perimeter` < 98.085”\n", + "
\n", + " \n", + " Measured Metric = 0.8\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean perimeter` >= 86.140 AND `mean perimeter` < 98.085\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`mean area` >= 518.300 AND `mean area` < 664.350”\n", + "
\n", + " \n", + " Measured Metric = 0.8\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean area` >= 518.300 AND `mean area` < 664.350\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`mean radius` >= 13.310 AND `mean radius` < 15.005”\n", + "
\n", + " \n", + " Measured Metric = 0.8\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean radius` >= 13.310 AND `mean radius` < 15.005\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`mean concave points` >= 0.048 AND `mean concave points` < 0.079”\n", + "
\n", + " \n", + " Measured Metric = 0.8\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean concave points` >= 0.048 AND `mean concave points` < 0.079\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`mean concavity` >= 0.078 AND `mean concavity` < 0.140”\n", + "
\n", + " \n", + " Measured Metric = 0.8\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean concavity` >= 0.078 AND `mean concavity` < 0.140\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`worst area` >= 610.200 AND `worst area` < 828.500”\n", + "
\n", + " \n", + " Measured Metric = 0.81818\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst area` >= 610.200 AND `worst area` < 828.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`fractal dimension error` < 0.003 AND `fractal dimension error` >= 0.002”\n", + "
\n", + " \n", + " Measured Metric = 0.83333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `fractal dimension error` < 0.003 AND `fractal dimension error` >= 0.002\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`worst concavity` >= 0.277 AND `worst concavity` < 0.415”\n", + "
\n", + " \n", + " Measured Metric = 0.83333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `worst concavity` >= 0.277 AND `worst concavity` < 0.415\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`compactness error` >= 0.020 AND `compactness error` < 0.030”\n", + "
\n", + " \n", + " Measured Metric = 0.84375\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `compactness error` >= 0.020 AND `compactness error` < 0.030\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`mean compactness` >= 0.099 AND `mean compactness` < 0.128”\n", + "
\n", + " \n", + " Measured Metric = 0.84375\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `mean compactness` >= 0.099 AND `mean compactness` < 0.128\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`concave points error` >= 0.009 AND `concave points error` < 0.015”\n", + "
\n", + " \n", + " Measured Metric = 0.875\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `concave points error` >= 0.009 AND `concave points error` < 0.015\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9179775280898876\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`perimeter error` < 2.151 AND `perimeter error` >= 1.505”\n", + "
\n", + " \n", + " Measured Metric = 0.88571\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `perimeter error` < 2.151 AND `perimeter error` >= 1.505\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`area error` < 40.745 AND `area error` >= 19.215”\n", + "
\n", + " \n", + " Measured Metric = 0.88889\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " breast_cancer_xgboost\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " breast_cancer\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `area error` < 40.745 AND `area error` >= 19.215\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9101398601398601\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2024-05-29 11:38:26,789 pid:50687 Thread-59 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,796 pid:50687 Thread-58 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,822 pid:50687 Thread-60 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,848 pid:50687 Thread-61 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,853 pid:50687 Thread-63 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,873 pid:50687 Thread-62 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:38:26,899 pid:50687 Thread-64 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n" + ] } ], "source": [ @@ -503,13 +5170,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Customize your suite by loading objects from the Giskard catalog" - ], + "id": "882f4638", "metadata": { "collapsed": false }, - "id": "882f4638" + "source": [ + "## Customize your suite by loading objects from the Giskard catalog" + ] }, { "cell_type": "markdown", @@ -538,124 +5205,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "id": "cf824254", - "metadata": {}, - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to: \n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ] - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - }, - "id": "b03d16b0b4dc2e6f" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - }, - "id": "eee658d69dd8d8c2" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - }, - "id": "ecde49539af0f699" - }, - { - "cell_type": "markdown", - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ], - "metadata": { - "collapsed": false - }, - "id": "984c06a4f3c1bb01" - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ], - "metadata": { - "collapsed": false - }, - "id": "a309531193bd89cc" - }, - { - "cell_type": "markdown", - "id": "193983c206c0103f", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - }, - "id": "e9dadbd02995199d" } ], "metadata": { @@ -674,7 +5223,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.11" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/churn_prediction_lgbm.ipynb b/docs/reference/notebooks/churn_prediction_lgbm.ipynb index 709fed0eb6..b56b11cd72 100644 --- a/docs/reference/notebooks/churn_prediction_lgbm.ipynb +++ b/docs/reference/notebooks/churn_prediction_lgbm.ipynb @@ -21,12 +21,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -41,7 +36,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2023-08-22T10:12:25.295802Z", @@ -96,7 +91,7 @@ "from sklearn.preprocessing import OneHotEncoder\n", "from sklearn.preprocessing import StandardScaler\n", "\n", - "from giskard import Dataset, Model, scan, GiskardClient, testing, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { @@ -112,11 +107,11 @@ "cell_type": "code", "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:45:06.756123Z", "start_time": "2023-11-08T22:45:06.722479Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -177,11 +172,11 @@ "cell_type": "code", "execution_count": 3, "metadata": { - "scrolled": false, "ExecuteTime": { "end_time": "2023-11-08T22:45:08.742520Z", "start_time": "2023-11-08T22:45:08.265673Z" - } + }, + "scrolled": false }, "outputs": [], "source": [ @@ -211,11 +206,11 @@ "cell_type": "code", "execution_count": 4, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:45:09.773474Z", "start_time": "2023-11-08T22:45:09.672374Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -238,13 +233,21 @@ "cell_type": "code", "execution_count": 5, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:45:10.845261Z", "start_time": "2023-11-08T22:45:10.797728Z" - } + }, + "collapsed": false }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2024-05-29 11:39:29,918 pid:51250 MainThread giskard.datasets.base INFO Your 'pandas.DataFrame' is successfully wrapped by Giskard's 'Dataset' wrapper class.\n" + ] + } + ], "source": [ "raw_data = pd.concat([X_test, Y_test], axis=1)\n", "giskard_dataset = Dataset(\n", @@ -275,11 +278,11 @@ "cell_type": "code", "execution_count": 6, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:45:12.378895Z", "start_time": "2023-11-08T22:45:12.356273Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -358,12 +361,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -391,16 +394,3238 @@ "cell_type": "code", "execution_count": 10, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:45:57.660834Z", "start_time": "2023-11-08T22:45:57.392784Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -421,145 +3646,1269 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", "execution_count": 11, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:46:08.319084Z", "start_time": "2023-11-08T22:46:07.195401Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Overconfidence on data slice “`TotalCharges` >= 3246.925”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}: \n", + "2024-05-29 11:43:15,920 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:15,923 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (506, 20) executed in 0:00:00.018539\n", + "Executed 'Overconfidence on data slice “`TotalCharges` >= 3246.925”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}: \n", " Test failed\n", " Metric: 0.56\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`InternetService` == \"DSL\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}: \n", + "2024-05-29 11:43:15,939 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:15,941 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (561, 20) executed in 0:00:00.010149\n", + "Executed 'Overconfidence on data slice “`InternetService` == \"DSL\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}: \n", " Test failed\n", " Metric: 0.51\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`OnlineBackup` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}: \n", + "2024-05-29 11:43:15,956 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:15,958 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (614, 20) executed in 0:00:00.010088\n", + "Executed 'Overconfidence on data slice “`OnlineBackup` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}: \n", " Test failed\n", " Metric: 0.46\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`OnlineSecurity` == \"No\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}: \n", + "2024-05-29 11:43:15,974 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:15,976 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (870, 20) executed in 0:00:00.010450\n", + "Executed 'Underconfidence on data slice “`OnlineSecurity` == \"No\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.02\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`Contract` == \"Month-to-month\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}: \n", + "2024-05-29 11:43:15,993 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:15,995 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (980, 20) executed in 0:00:00.011023\n", + "Executed 'Underconfidence on data slice “`Contract` == \"Month-to-month\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.02\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`Dependents` == \"No\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}: \n", + "2024-05-29 11:43:16,017 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,021 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1248, 20) executed in 0:00:00.014565\n", + "Executed 'Underconfidence on data slice “`Dependents` == \"No\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.02\n", " \n", " \n", - "Executed 'Recall on data slice “`Contract` == \"One year\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,035 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,037 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (376, 20) executed in 0:00:00.009567\n", + "Executed 'Recall on data slice “`Contract` == \"One year\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.0\n", " \n", " \n", - "Executed 'Recall on data slice “`tenure` >= 44.500 AND `tenure` < 70.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,054 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,055 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (497, 20) executed in 0:00:00.008861\n", + "Executed 'Recall on data slice “`tenure` >= 44.500 AND `tenure` < 70.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.06\n", " \n", " \n", - "Executed 'Recall on data slice “`InternetService` == \"No\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,071 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,074 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (405, 20) executed in 0:00:00.009695\n", + "Executed 'Recall on data slice “`InternetService` == \"No\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.08\n", " \n", " \n", - "Executed 'Recall on data slice “`OnlineSecurity` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,092 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,093 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (405, 20) executed in 0:00:00.008504\n", + "Executed 'Recall on data slice “`OnlineSecurity` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.08\n", " \n", " \n", - "Executed 'Recall on data slice “`OnlineBackup` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,109 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,111 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (405, 20) executed in 0:00:00.008597\n", + "Executed 'Recall on data slice “`OnlineBackup` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.08\n", " \n", " \n", - "Executed 'Recall on data slice “`DeviceProtection` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,125 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,128 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (405, 20) executed in 0:00:00.008774\n", + "Executed 'Recall on data slice “`DeviceProtection` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.08\n", " \n", " \n", - "Executed 'Recall on data slice “`TechSupport` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,143 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,145 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (405, 20) executed in 0:00:00.008562\n", + "Executed 'Recall on data slice “`TechSupport` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.08\n", " \n", " \n", - "Executed 'Recall on data slice “`StreamingTV` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,161 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,164 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (405, 20) executed in 0:00:00.010419\n", + "Executed 'Recall on data slice “`StreamingTV` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.08\n", " \n", " \n", - "Executed 'Recall on data slice “`StreamingMovies` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,179 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,182 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (405, 20) executed in 0:00:00.010005\n", + "Executed 'Recall on data slice “`StreamingMovies` == \"No internet service\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.08\n", " \n", " \n", - "Executed 'Recall on data slice “`MonthlyCharges` < 20.775”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,196 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,198 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (296, 20) executed in 0:00:00.008635\n", + "Executed 'Recall on data slice “`MonthlyCharges` < 20.775”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.1\n", " \n", " \n", - "Executed 'Recall on data slice “`TechSupport` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,214 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,216 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (472, 20) executed in 0:00:00.009999\n", + "Executed 'Recall on data slice “`TechSupport` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.21\n", " \n", " \n", - "Executed 'Recall on data slice “`OnlineSecurity` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,231 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,233 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (483, 20) executed in 0:00:00.009687\n", + "Executed 'Recall on data slice “`OnlineSecurity` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.21\n", " \n", " \n", - "Executed 'Recall on data slice “`PaymentMethod` == \"Credit card\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,256 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,268 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (368, 20) executed in 0:00:00.025869\n", + "Executed 'Recall on data slice “`PaymentMethod` == \"Credit card\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.28\n", " \n", " \n", - "Executed 'Recall on data slice “`InternetService` == \"DSL\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,297 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,305 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (561, 20) executed in 0:00:00.017681\n", + "Executed 'Recall on data slice “`InternetService` == \"DSL\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.32\n", " \n", " \n", - "Executed 'Recall on data slice “`Dependents` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", + "2024-05-29 11:43:16,332 pid:51250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'} to {'gender': 'object', 'SeniorCitizen': 'int64', 'Partner': 'object', 'Dependents': 'object', 'tenure': 'int64', 'PhoneService': 'object', 'MultipleLines': 'object', 'InternetService': 'object', 'OnlineSecurity': 'object', 'OnlineBackup': 'object', 'DeviceProtection': 'object', 'TechSupport': 'object', 'StreamingTV': 'object', 'StreamingMovies': 'object', 'Contract': 'object', 'PaperlessBilling': 'object', 'PaymentMethod': 'object', 'MonthlyCharges': 'float64', 'TotalCharges': 'float64'}\n", + "2024-05-29 11:43:16,340 pid:51250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (510, 20) executed in 0:00:00.020697\n", + "Executed 'Recall on data slice “`Dependents` == \"Yes\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}: \n", " Test failed\n", " Metric: 0.33\n", " \n", - " \n" + " \n", + "2024-05-29 11:43:16,348 pid:51250 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 11:43:16,348 pid:51250 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 11:43:16,349 pid:51250 MainThread giskard.core.suite INFO Overconfidence on data slice “`TotalCharges` >= 3246.925” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}): {failed, metric=0.5568181818181818}\n", + "2024-05-29 11:43:16,349 pid:51250 MainThread giskard.core.suite INFO Overconfidence on data slice “`InternetService` == \"DSL\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}): {failed, metric=0.5053763440860215}\n", + "2024-05-29 11:43:16,349 pid:51250 MainThread giskard.core.suite INFO Overconfidence on data slice “`OnlineBackup` == \"Yes\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4486033519553073, 'p_threshold': 0.5}): {failed, metric=0.45588235294117646}\n", + "2024-05-29 11:43:16,349 pid:51250 MainThread giskard.core.suite INFO Underconfidence on data slice “`OnlineSecurity` == \"No\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}): {failed, metric=0.02413793103448276}\n", + "2024-05-29 11:43:16,350 pid:51250 MainThread giskard.core.suite INFO Underconfidence on data slice “`Contract` == \"Month-to-month\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}): {failed, metric=0.022448979591836733}\n", + "2024-05-29 11:43:16,350 pid:51250 MainThread giskard.core.suite INFO Underconfidence on data slice “`Dependents` == \"No\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.014391353811149032, 'p_threshold': 0.95}): {failed, metric=0.016826923076923076}\n", + "2024-05-29 11:43:16,350 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`Contract` == \"One year\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.0}\n", + "2024-05-29 11:43:16,351 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`tenure` >= 44.500 AND `tenure` < 70.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.05970149253731343}\n", + "2024-05-29 11:43:16,351 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`InternetService` == \"No\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.07692307692307693}\n", + "2024-05-29 11:43:16,351 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`OnlineSecurity` == \"No internet service\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.07692307692307693}\n", + "2024-05-29 11:43:16,352 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`OnlineBackup` == \"No internet service\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.07692307692307693}\n", + "2024-05-29 11:43:16,352 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`DeviceProtection` == \"No internet service\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.07692307692307693}\n", + "2024-05-29 11:43:16,352 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`TechSupport` == \"No internet service\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.07692307692307693}\n", + "2024-05-29 11:43:16,353 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`StreamingTV` == \"No internet service\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.07692307692307693}\n", + "2024-05-29 11:43:16,353 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`StreamingMovies` == \"No internet service\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.07692307692307693}\n", + "2024-05-29 11:43:16,353 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`MonthlyCharges` < 20.775” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.10344827586206896}\n", + "2024-05-29 11:43:16,353 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`TechSupport` == \"Yes\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.2054794520547945}\n", + "2024-05-29 11:43:16,353 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`OnlineSecurity` == \"Yes\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.2125}\n", + "2024-05-29 11:43:16,354 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`PaymentMethod` == \"Credit card\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.2777777777777778}\n", + "2024-05-29 11:43:16,354 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`InternetService` == \"DSL\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.3181818181818182}\n", + "2024-05-29 11:43:16,354 pid:51250 MainThread giskard.core.suite INFO Recall on data slice “`Dependents` == \"Yes\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.49131679389312977}): {failed, metric=0.32558139534883723}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`TotalCharges` >= 3246.925”\n
\n \n Measured Metric = 0.55682\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `TotalCharges` >= 3246.925\n
\n \n
\n threshold\n 0.4486033519553073\n
\n \n
\n p_threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`InternetService` == "DSL"”\n
\n \n Measured Metric = 0.50538\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `InternetService` == "DSL"\n
\n \n
\n threshold\n 0.4486033519553073\n
\n \n
\n p_threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`OnlineBackup` == "Yes"”\n
\n \n Measured Metric = 0.45588\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `OnlineBackup` == "Yes"\n
\n \n
\n threshold\n 0.4486033519553073\n
\n \n
\n p_threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Underconfidence on data slice “`OnlineSecurity` == "No"”\n
\n \n Measured Metric = 0.02414\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `OnlineSecurity` == "No"\n
\n \n
\n threshold\n 0.014391353811149032\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Underconfidence on data slice “`Contract` == "Month-to-month"”\n
\n \n Measured Metric = 0.02245\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `Contract` == "Month-to-month"\n
\n \n
\n threshold\n 0.014391353811149032\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Underconfidence on data slice “`Dependents` == "No"”\n
\n \n Measured Metric = 0.01683\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `Dependents` == "No"\n
\n \n
\n threshold\n 0.014391353811149032\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`Contract` == "One year"”\n
\n \n Measured Metric = 0.0\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `Contract` == "One year"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`tenure` >= 44.500 AND `tenure` < 70.500”\n
\n \n Measured Metric = 0.0597\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `tenure` >= 44.500 AND `tenure` < 70.500\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`InternetService` == "No"”\n
\n \n Measured Metric = 0.07692\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `InternetService` == "No"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`OnlineSecurity` == "No internet service"”\n
\n \n Measured Metric = 0.07692\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `OnlineSecurity` == "No internet service"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`OnlineBackup` == "No internet service"”\n
\n \n Measured Metric = 0.07692\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `OnlineBackup` == "No internet service"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`DeviceProtection` == "No internet service"”\n
\n \n Measured Metric = 0.07692\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `DeviceProtection` == "No internet service"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`TechSupport` == "No internet service"”\n
\n \n Measured Metric = 0.07692\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `TechSupport` == "No internet service"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`StreamingTV` == "No internet service"”\n
\n \n Measured Metric = 0.07692\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `StreamingTV` == "No internet service"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`StreamingMovies` == "No internet service"”\n
\n \n Measured Metric = 0.07692\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `StreamingMovies` == "No internet service"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`MonthlyCharges` < 20.775”\n
\n \n Measured Metric = 0.10345\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `MonthlyCharges` < 20.775\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`TechSupport` == "Yes"”\n
\n \n Measured Metric = 0.20548\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `TechSupport` == "Yes"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`OnlineSecurity` == "Yes"”\n
\n \n Measured Metric = 0.2125\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `OnlineSecurity` == "Yes"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`PaymentMethod` == "Credit card"”\n
\n \n Measured Metric = 0.27778\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `PaymentMethod` == "Credit card"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`InternetService` == "DSL"”\n
\n \n Measured Metric = 0.31818\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `InternetService` == "DSL"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`Dependents` == "Yes"”\n
\n \n Measured Metric = 0.32558\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n dbe1aaf2-396d-4fc2-8c13-1128bb44cf24\n
\n \n
\n dataset\n Churn classification dataset\n
\n \n
\n slicing_function\n `Dependents` == "Yes"\n
\n \n
\n threshold\n 0.49131679389312977\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`TotalCharges` >= 3246.925”\n", + "
\n", + " \n", + " Measured Metric = 0.55682\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `TotalCharges` >= 3246.925\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.4486033519553073\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`InternetService` == "DSL"”\n", + "
\n", + " \n", + " Measured Metric = 0.50538\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `InternetService` == "DSL"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.4486033519553073\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`OnlineBackup` == "Yes"”\n", + "
\n", + " \n", + " Measured Metric = 0.45588\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `OnlineBackup` == "Yes"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.4486033519553073\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`OnlineSecurity` == "No"”\n", + "
\n", + " \n", + " Measured Metric = 0.02414\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `OnlineSecurity` == "No"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.014391353811149032\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`Contract` == "Month-to-month"”\n", + "
\n", + " \n", + " Measured Metric = 0.02245\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Contract` == "Month-to-month"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.014391353811149032\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`Dependents` == "No"”\n", + "
\n", + " \n", + " Measured Metric = 0.01683\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Dependents` == "No"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.014391353811149032\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`Contract` == "One year"”\n", + "
\n", + " \n", + " Measured Metric = 0.0\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Contract` == "One year"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`tenure` >= 44.500 AND `tenure` < 70.500”\n", + "
\n", + " \n", + " Measured Metric = 0.0597\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `tenure` >= 44.500 AND `tenure` < 70.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`InternetService` == "No"”\n", + "
\n", + " \n", + " Measured Metric = 0.07692\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `InternetService` == "No"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`OnlineSecurity` == "No internet service"”\n", + "
\n", + " \n", + " Measured Metric = 0.07692\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `OnlineSecurity` == "No internet service"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`OnlineBackup` == "No internet service"”\n", + "
\n", + " \n", + " Measured Metric = 0.07692\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `OnlineBackup` == "No internet service"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`DeviceProtection` == "No internet service"”\n", + "
\n", + " \n", + " Measured Metric = 0.07692\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `DeviceProtection` == "No internet service"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`TechSupport` == "No internet service"”\n", + "
\n", + " \n", + " Measured Metric = 0.07692\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `TechSupport` == "No internet service"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`StreamingTV` == "No internet service"”\n", + "
\n", + " \n", + " Measured Metric = 0.07692\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `StreamingTV` == "No internet service"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`StreamingMovies` == "No internet service"”\n", + "
\n", + " \n", + " Measured Metric = 0.07692\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `StreamingMovies` == "No internet service"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`MonthlyCharges` < 20.775”\n", + "
\n", + " \n", + " Measured Metric = 0.10345\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `MonthlyCharges` < 20.775\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`TechSupport` == "Yes"”\n", + "
\n", + " \n", + " Measured Metric = 0.20548\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `TechSupport` == "Yes"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`OnlineSecurity` == "Yes"”\n", + "
\n", + " \n", + " Measured Metric = 0.2125\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `OnlineSecurity` == "Yes"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`PaymentMethod` == "Credit card"”\n", + "
\n", + " \n", + " Measured Metric = 0.27778\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `PaymentMethod` == "Credit card"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`InternetService` == "DSL"”\n", + "
\n", + " \n", + " Measured Metric = 0.31818\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `InternetService` == "DSL"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`Dependents` == "Yes"”\n", + "
\n", + " \n", + " Measured Metric = 0.32558\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Churn classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Churn classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Dependents` == "Yes"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.49131679389312977\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2024-05-29 11:43:16,426 pid:51250 Thread-36 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:43:16,454 pid:51250 Thread-37 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n", + "2024-05-29 11:43:16,485 pid:51250 Thread-38 (_track) urllib3.connectionpool WARNING Connection pool is full, discarding connection: api.mixpanel.com. Connection pool size: 10\n" + ] } ], "source": [ @@ -596,118 +4945,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to: \n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - " \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -737,7 +4974,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.15" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/credit_scoring.ipynb b/docs/reference/notebooks/credit_scoring.ipynb index 2d6bcb2ff4..db450b2446 100644 --- a/docs/reference/notebooks/credit_scoring.ipynb +++ b/docs/reference/notebooks/credit_scoring.ipynb @@ -21,12 +21,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -41,7 +36,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2023-08-21T12:06:20.563432Z", @@ -67,11 +62,11 @@ "cell_type": "code", "execution_count": 1, "metadata": { - "collapsed": true, "ExecuteTime": { "end_time": "2023-11-08T22:50:16.461436Z", "start_time": "2023-11-08T22:50:09.622133Z" - } + }, + "collapsed": true }, "outputs": [], "source": [ @@ -84,7 +79,7 @@ "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n", "\n", - "from giskard import Model, Dataset, scan, testing, GiskardClient, Suite" + "from giskard import Model, Dataset, scan, testing" ] }, { @@ -100,11 +95,11 @@ "cell_type": "code", "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:50:22.598805Z", "start_time": "2023-11-08T22:50:22.551813Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -164,11 +159,11 @@ "cell_type": "code", "execution_count": 3, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:50:24.243291Z", "start_time": "2023-11-08T22:50:23.880752Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -188,11 +183,11 @@ "cell_type": "code", "execution_count": 4, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:50:24.772138Z", "start_time": "2023-11-08T22:50:24.740667Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -212,13 +207,13 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:50:25.702983Z", "start_time": "2023-11-08T22:50:25.668475Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -255,11 +250,11 @@ "cell_type": "code", "execution_count": 6, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:50:26.994317Z", "start_time": "2023-11-08T22:50:26.960393Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -343,12 +338,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -376,16 +371,2422 @@ "cell_type": "code", "execution_count": 10, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-08T22:51:12.440487Z", "start_time": "2023-11-08T22:51:12.179232Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -397,12 +2798,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Generate comprehensive test suites automatically for your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Generate comprehensive test suites automatically for your model" + ] }, { "cell_type": "markdown", @@ -418,52 +2819,500 @@ { "cell_type": "code", "execution_count": 11, + "metadata": { + "ExecuteTime": { + "end_time": "2023-11-08T22:51:26.616126Z", + "start_time": "2023-11-08T22:51:26.248461Z" + }, + "collapsed": false + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Precision on data slice “`other_installment_plans` == \"bank\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", + "2024-05-29 11:45:25,982 pid:52370 MainThread giskard.datasets.base INFO Casting dataframe columns from {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'} to {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'}\n", + "2024-05-29 11:45:25,986 pid:52370 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (32, 22) executed in 0:00:00.014548\n", + "Executed 'Precision on data slice “`other_installment_plans` == \"bank\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", " Test failed\n", " Metric: 0.6\n", " \n", " \n", - "Executed 'Precision on data slice “`account_check_status` == \"0 <= ... < 200 DM\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", + "2024-05-29 11:45:26,003 pid:52370 MainThread giskard.datasets.base INFO Casting dataframe columns from {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'} to {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'}\n", + "2024-05-29 11:45:26,006 pid:52370 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (58, 22) executed in 0:00:00.011488\n", + "Executed 'Precision on data slice “`account_check_status` == \"0 <= ... < 200 DM\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", " Test failed\n", " Metric: 0.6\n", " \n", " \n", - "Executed 'Precision on data slice “`present_employment_since` == \"... < 1 year\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", + "2024-05-29 11:45:26,019 pid:52370 MainThread giskard.datasets.base INFO Casting dataframe columns from {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'} to {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'}\n", + "2024-05-29 11:45:26,021 pid:52370 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (37, 22) executed in 0:00:00.008733\n", + "Executed 'Precision on data slice “`present_employment_since` == \"... < 1 year\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", " Test failed\n", " Metric: 0.65\n", " \n", " \n", - "Executed 'Recall on data slice “`personal_status` == \"divorced\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8617857142857143}: \n", + "2024-05-29 11:45:26,033 pid:52370 MainThread giskard.datasets.base INFO Casting dataframe columns from {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'} to {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'}\n", + "2024-05-29 11:45:26,036 pid:52370 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (68, 22) executed in 0:00:00.008518\n", + "Executed 'Recall on data slice “`personal_status` == \"divorced\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8617857142857143}: \n", " Test failed\n", " Metric: 0.8\n", " \n", " \n", - "Executed 'Precision on data slice “`duration_in_month` >= 16.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", + "2024-05-29 11:45:26,050 pid:52370 MainThread giskard.datasets.base INFO Casting dataframe columns from {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'} to {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'}\n", + "2024-05-29 11:45:26,054 pid:52370 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (112, 22) executed in 0:00:00.009998\n", + "Executed 'Precision on data slice “`duration_in_month` >= 16.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", " Test failed\n", " Metric: 0.71\n", " \n", " \n", - "Executed 'Precision on data slice “`property` == \"if not A121/A122 : car or other, not in attribute 6\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", + "2024-05-29 11:45:26,072 pid:52370 MainThread giskard.datasets.base INFO Casting dataframe columns from {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'} to {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'}\n", + "2024-05-29 11:45:26,074 pid:52370 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (69, 22) executed in 0:00:00.012084\n", + "Executed 'Precision on data slice “`property` == \"if not A121/A122 : car or other, not in attribute 6\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", " Test failed\n", " Metric: 0.72\n", " \n", " \n", - "Executed 'Precision on data slice “`sex` == \"female\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", + "2024-05-29 11:45:26,087 pid:52370 MainThread giskard.datasets.base INFO Casting dataframe columns from {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'} to {'account_check_status': 'object', 'duration_in_month': 'int64', 'credit_history': 'object', 'purpose': 'object', 'credit_amount': 'int64', 'savings': 'object', 'present_employment_since': 'object', 'installment_as_income_perc': 'int64', 'sex': 'object', 'personal_status': 'object', 'other_debtors': 'object', 'present_residence_since': 'int64', 'property': 'object', 'age': 'int64', 'other_installment_plans': 'object', 'housing': 'object', 'credits_this_bank': 'int64', 'job': 'object', 'people_under_maintenance': 'int64', 'telephone': 'object', 'foreign_worker': 'object'}\n", + "2024-05-29 11:45:26,089 pid:52370 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (54, 22) executed in 0:00:00.009407\n", + "Executed 'Precision on data slice “`sex` == \"female\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}: \n", " Test failed\n", " Metric: 0.74\n", " \n", - " \n" + " \n", + "2024-05-29 11:45:26,093 pid:52370 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 11:45:26,093 pid:52370 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 11:45:26,094 pid:52370 MainThread giskard.core.suite INFO Precision on data slice “`other_installment_plans` == \"bank\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}): {failed, metric=0.6}\n", + "2024-05-29 11:45:26,094 pid:52370 MainThread giskard.core.suite INFO Precision on data slice “`account_check_status` == \"0 <= ... < 200 DM\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}): {failed, metric=0.6046511627906976}\n", + "2024-05-29 11:45:26,094 pid:52370 MainThread giskard.core.suite INFO Precision on data slice “`present_employment_since` == \"... < 1 year\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}): {failed, metric=0.6521739130434783}\n", + "2024-05-29 11:45:26,094 pid:52370 MainThread giskard.core.suite INFO Recall on data slice “`personal_status` == \"divorced\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8617857142857143}): {failed, metric=0.8048780487804879}\n", + "2024-05-29 11:45:26,095 pid:52370 MainThread giskard.core.suite INFO Precision on data slice “`duration_in_month` >= 16.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}): {failed, metric=0.7125}\n", + "2024-05-29 11:45:26,095 pid:52370 MainThread giskard.core.suite INFO Precision on data slice “`property` == \"if not A121/A122 : car or other, not in attribute 6\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}): {failed, metric=0.7192982456140351}\n", + "2024-05-29 11:45:26,095 pid:52370 MainThread giskard.core.suite INFO Precision on data slice “`sex` == \"female\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7540625}): {failed, metric=0.7368421052631579}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Precision on data slice “`other_installment_plans` == "bank"”\n
\n \n Measured Metric = 0.6\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n a4d8f299-351e-4722-8176-9b37fa35a9e8\n
\n \n
\n dataset\n German credit scoring dataset\n
\n \n
\n slicing_function\n `other_installment_plans` == "bank"\n
\n \n
\n threshold\n 0.7540625\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`account_check_status` == "0 <= ... < 200 DM"”\n
\n \n Measured Metric = 0.60465\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n a4d8f299-351e-4722-8176-9b37fa35a9e8\n
\n \n
\n dataset\n German credit scoring dataset\n
\n \n
\n slicing_function\n `account_check_status` == "0 <= ... < 200 DM"\n
\n \n
\n threshold\n 0.7540625\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`present_employment_since` == "... < 1 year"”\n
\n \n Measured Metric = 0.65217\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n a4d8f299-351e-4722-8176-9b37fa35a9e8\n
\n \n
\n dataset\n German credit scoring dataset\n
\n \n
\n slicing_function\n `present_employment_since` == "... < 1 year"\n
\n \n
\n threshold\n 0.7540625\n
\n \n
\n
\n \n \n
\n Test Recall on data slice “`personal_status` == "divorced"”\n
\n \n Measured Metric = 0.80488\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n a4d8f299-351e-4722-8176-9b37fa35a9e8\n
\n \n
\n dataset\n German credit scoring dataset\n
\n \n
\n slicing_function\n `personal_status` == "divorced"\n
\n \n
\n threshold\n 0.8617857142857143\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`duration_in_month` >= 16.500”\n
\n \n Measured Metric = 0.7125\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n a4d8f299-351e-4722-8176-9b37fa35a9e8\n
\n \n
\n dataset\n German credit scoring dataset\n
\n \n
\n slicing_function\n `duration_in_month` >= 16.500\n
\n \n
\n threshold\n 0.7540625\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`property` == "if not A121/A122 : car or other, not in attribute 6"”\n
\n \n Measured Metric = 0.7193\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n a4d8f299-351e-4722-8176-9b37fa35a9e8\n
\n \n
\n dataset\n German credit scoring dataset\n
\n \n
\n slicing_function\n `property` == "if not A121/A122 : car or other, not in attribute 6"\n
\n \n
\n threshold\n 0.7540625\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`sex` == "female"”\n
\n \n Measured Metric = 0.73684\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n a4d8f299-351e-4722-8176-9b37fa35a9e8\n
\n \n
\n dataset\n German credit scoring dataset\n
\n \n
\n slicing_function\n `sex` == "female"\n
\n \n
\n threshold\n 0.7540625\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`other_installment_plans` == "bank"”\n", + "
\n", + " \n", + " Measured Metric = 0.6\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Credit scoring classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " German credit scoring dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `other_installment_plans` == "bank"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7540625\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`account_check_status` == "0 <= ... < 200 DM"”\n", + "
\n", + " \n", + " Measured Metric = 0.60465\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Credit scoring classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " German credit scoring dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `account_check_status` == "0 <= ... < 200 DM"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7540625\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`present_employment_since` == "... < 1 year"”\n", + "
\n", + " \n", + " Measured Metric = 0.65217\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Credit scoring classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " German credit scoring dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `present_employment_since` == "... < 1 year"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7540625\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`personal_status` == "divorced"”\n", + "
\n", + " \n", + " Measured Metric = 0.80488\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Credit scoring classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " German credit scoring dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `personal_status` == "divorced"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.8617857142857143\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`duration_in_month` >= 16.500”\n", + "
\n", + " \n", + " Measured Metric = 0.7125\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Credit scoring classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " German credit scoring dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `duration_in_month` >= 16.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7540625\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`property` == "if not A121/A122 : car or other, not in attribute 6"”\n", + "
\n", + " \n", + " Measured Metric = 0.7193\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Credit scoring classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " German credit scoring dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `property` == "if not A121/A122 : car or other, not in attribute 6"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7540625\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`sex` == "female"”\n", + "
\n", + " \n", + " Measured Metric = 0.73684\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Credit scoring classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " German credit scoring dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `sex` == "female"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7540625\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 11, "metadata": {}, @@ -473,17 +3322,13 @@ "source": [ "test_suite = results.generate_test_suite(\"My first test suite\")\n", "test_suite.run()" - ], - "metadata": { - "collapsed": false, - "ExecuteTime": { - "end_time": "2023-11-08T22:51:26.616126Z", - "start_time": "2023-11-08T22:51:26.248461Z" - } - } + ] }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Customize your suite by loading objects from the Giskard catalog\n", "\n", @@ -496,79 +3341,6 @@ "To create custom tests, refer to [this page](https://docs.giskard.ai/en/stable/open_source/customize_tests/test_model/index.html).\n", "\n", "For demo purposes, we will load a simple unit test (test_f1) that checks if the test F1 score is above the given threshold. For more examples of tests and functions, refer to the Giskard catalog." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" - ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." ] }, { @@ -579,49 +3351,7 @@ }, "outputs": [], "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - " \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" + "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] } ], @@ -634,14 +3364,14 @@ "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.6" + "pygments_lexer": "ipython3", + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/drug_classification_sklearn.ipynb b/docs/reference/notebooks/drug_classification_sklearn.ipynb index 1f4109015c..6c443f487b 100644 --- a/docs/reference/notebooks/drug_classification_sklearn.ipynb +++ b/docs/reference/notebooks/drug_classification_sklearn.ipynb @@ -22,12 +22,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -43,7 +38,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": null, "id": "85a76ae027fad887", "metadata": { "ExecuteTime": { @@ -69,7 +64,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "5fdae2be34577a32", "metadata": { "collapsed": false @@ -88,7 +83,7 @@ "from sklearn.model_selection import train_test_split\n", "from imblearn.pipeline import Pipeline as PipelineImb\n", "\n", - "from giskard import Dataset, Model, scan, GiskardClient, testing, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { @@ -106,11 +101,11 @@ "execution_count": 2, "id": "d44430add2918aa1", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2024-02-09T09:29:15.513819Z", "start_time": "2024-02-09T09:29:15.470284Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -155,11 +150,11 @@ "execution_count": 3, "id": "5a2fbb53dd96b195", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2024-02-09T09:29:16.892145Z", "start_time": "2024-02-09T09:29:16.870124Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -204,9 +199,13 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "id": "a1887adb", "metadata": { + "ExecuteTime": { + "end_time": "2024-02-09T09:29:17.524094Z", + "start_time": "2024-02-09T09:29:17.478053Z" + }, "execution": { "iopub.execute_input": "2022-05-04T02:53:16.032035Z", "iopub.status.busy": "2022-05-04T02:53:16.030941Z", @@ -222,21 +221,9 @@ "start_time": "2022-05-04T02:53:15.970178", "status": "completed" }, - "tags": [], - "ExecuteTime": { - "end_time": "2024-02-09T09:29:17.524094Z", - "start_time": "2024-02-09T09:29:17.478053Z" - } + "tags": [] }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Data was loaded!\n" - ] - } - ], + "outputs": [], "source": [ "df_drug = load_data()\n", "df_drug = bin_numerical(df_drug)" @@ -257,11 +244,11 @@ "execution_count": 5, "id": "7c32c64979960c7d", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2024-02-09T09:29:19.259540Z", "start_time": "2024-02-09T09:29:18.975922Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -282,14 +269,14 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "id": "3c6c6bea2652fe95", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2024-02-09T09:29:20.755641Z", "start_time": "2024-02-09T09:29:20.712569Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -322,13 +309,13 @@ }, { "cell_type": "markdown", - "source": [ - "### Build estimator" - ], + "id": "a753c44d4e184034", "metadata": { "collapsed": false }, - "id": "a753c44d4e184034" + "source": [ + "### Build estimator" + ] }, { "cell_type": "code", @@ -396,13 +383,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], + "id": "cba9f80f09e85818", "metadata": { "collapsed": false }, - "id": "cba9f80f09e85818" + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -430,19 +417,1127 @@ }, { "cell_type": "code", - "execution_count": 65, + "execution_count": 10, "id": "82db2acb6c2ae6dd", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2024-02-09T10:02:51.276834Z", "start_time": "2024-02-09T10:02:50.767947Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -464,60 +1559,363 @@ }, { "cell_type": "markdown", + "id": "2047c63c4654ad4c", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - }, - "id": "2047c63c4654ad4c" + ] }, { "cell_type": "code", - "execution_count": 66, + "execution_count": 11, "id": "f44d26a78bda617e", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2024-02-09T10:11:54.015092Z", "start_time": "2024-02-09T10:11:53.880966Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Precision on data slice “`Age` == \"30s\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", + "2024-05-29 11:46:34,990 pid:52758 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'} to {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'}\n", + "2024-05-29 11:46:34,993 pid:52758 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (22, 6) executed in 0:00:00.009771\n", + "Executed 'Precision on data slice “`Age` == \"30s\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", " Test failed\n", " Metric: 0.68\n", " \n", " \n", - "Executed 'Precision on data slice “`BP` == \"NORMAL\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", + "2024-05-29 11:46:35,010 pid:52758 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'} to {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'}\n", + "2024-05-29 11:46:35,011 pid:52758 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (33, 6) executed in 0:00:00.007762\n", + "Executed 'Precision on data slice “`BP` == \"NORMAL\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", " Test failed\n", " Metric: 0.73\n", " \n", " \n", - "Executed 'Precision on data slice “`Na_to_K` == \"10-20\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", + "2024-05-29 11:46:35,023 pid:52758 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'} to {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'}\n", + "2024-05-29 11:46:35,025 pid:52758 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (59, 6) executed in 0:00:00.006482\n", + "Executed 'Precision on data slice “`Na_to_K` == \"10-20\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", " Test failed\n", " Metric: 0.73\n", " \n", " \n", - "Executed 'Precision on data slice “`Cholesterol` == \"HIGH\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", + "2024-05-29 11:46:35,034 pid:52758 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'} to {'Age': 'category', 'Sex': 'object', 'BP': 'object', 'Cholesterol': 'object', 'Na_to_K': 'category'}\n", + "2024-05-29 11:46:35,036 pid:52758 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (53, 6) executed in 0:00:00.006638\n", + "Executed 'Precision on data slice “`Cholesterol` == \"HIGH\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}: \n", " Test failed\n", " Metric: 0.75\n", " \n", - " \n" + " \n", + "2024-05-29 11:46:35,039 pid:52758 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 11:46:35,040 pid:52758 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 11:46:35,040 pid:52758 MainThread giskard.core.suite INFO Precision on data slice “`Age` == \"30s\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}): {failed, metric=0.6818181818181818}\n", + "2024-05-29 11:46:35,040 pid:52758 MainThread giskard.core.suite INFO Precision on data slice “`BP` == \"NORMAL\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}): {failed, metric=0.7272727272727273}\n", + "2024-05-29 11:46:35,041 pid:52758 MainThread giskard.core.suite INFO Precision on data slice “`Na_to_K` == \"10-20\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}): {failed, metric=0.7288135593220338}\n", + "2024-05-29 11:46:35,041 pid:52758 MainThread giskard.core.suite INFO Precision on data slice “`Cholesterol` == \"HIGH\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.76}): {failed, metric=0.7547169811320755}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Precision on data slice “`Age` == "30s"”\n
\n \n Measured Metric = 0.68182\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n Drug classifier\n
\n \n
\n dataset\n Drug classification dataset\n
\n \n
\n slicing_function\n `Age` == "30s"\n
\n \n
\n threshold\n 0.76\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`BP` == "NORMAL"”\n
\n \n Measured Metric = 0.72727\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n Drug classifier\n
\n \n
\n dataset\n Drug classification dataset\n
\n \n
\n slicing_function\n `BP` == "NORMAL"\n
\n \n
\n threshold\n 0.76\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`Na_to_K` == "10-20"”\n
\n \n Measured Metric = 0.72881\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n Drug classifier\n
\n \n
\n dataset\n Drug classification dataset\n
\n \n
\n slicing_function\n `Na_to_K` == "10-20"\n
\n \n
\n threshold\n 0.76\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`Cholesterol` == "HIGH"”\n
\n \n Measured Metric = 0.75472\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n Drug classifier\n
\n \n
\n dataset\n Drug classification dataset\n
\n \n
\n slicing_function\n `Cholesterol` == "HIGH"\n
\n \n
\n threshold\n 0.76\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Age` == "30s"”\n", + "
\n", + " \n", + " Measured Metric = 0.68182\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Drug classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Drug classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Age` == "30s"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.76\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`BP` == "NORMAL"”\n", + "
\n", + " \n", + " Measured Metric = 0.72727\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Drug classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Drug classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `BP` == "NORMAL"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.76\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Na_to_K` == "10-20"”\n", + "
\n", + " \n", + " Measured Metric = 0.72881\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Drug classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Drug classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Na_to_K` == "10-20"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.76\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Cholesterol` == "HIGH"”\n", + "
\n", + " \n", + " Measured Metric = 0.75472\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Drug classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Drug classification dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Cholesterol` == "HIGH"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.76\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 66, + "execution_count": 11, "metadata": {}, "output_type": "execute_result" } @@ -558,126 +1956,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=wrapped_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - }, - "id": "850f69c9e106c2c7" - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - }, - "id": "428e6ea983a1c8a5" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - }, - "id": "d2987c5a0a27a36a" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - }, - "id": "9f7b1a4e50f63aad" - }, - { - "cell_type": "markdown", - "id": "f2c270bda4037820", - "metadata": { - "collapsed": false - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "95186436fe201810", - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "id": "5e6a3c1e12f8cedd", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - " \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - }, - "id": "e3d431fdd4fcc407" } ], "metadata": { @@ -696,7 +1974,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.10" + "version": "3.10.14" }, "papermill": { "default_parameters": {}, diff --git a/docs/reference/notebooks/enron_email_classification_sklearn.ipynb b/docs/reference/notebooks/enron_email_classification_sklearn.ipynb index 24647b3874..cc705937be 100644 --- a/docs/reference/notebooks/enron_email_classification_sklearn.ipynb +++ b/docs/reference/notebooks/enron_email_classification_sklearn.ipynb @@ -21,12 +21,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -62,7 +57,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 1, "id": "7d960163", "metadata": { "ExecuteTime": { @@ -90,22 +85,30 @@ "from sklearn.feature_extraction.text import CountVectorizer\n", "from sklearn.feature_extraction.text import TfidfTransformer\n", "\n", - "from giskard import Dataset, Model, scan, testing, Suite, GiskardClient" + "from giskard import Dataset, Model, scan, testing" ] }, { "cell_type": "markdown", - "source": [ - "## Define constants" - ], + "id": "80172d2eb14941f5", "metadata": { "collapsed": false }, - "id": "80172d2eb14941f5" + "source": [ + "## Define constants" + ] }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 2, + "id": "4e5025e1dfa0343f", + "metadata": { + "ExecuteTime": { + "end_time": "2023-11-09T11:24:20.678786Z", + "start_time": "2023-11-09T11:24:20.085220Z" + }, + "collapsed": false + }, "outputs": [], "source": [ "TEXT_COLUMN = \"Content\"\n", @@ -132,15 +135,7 @@ " 13: 'trip reports'}\n", "\n", "LABEL_CAT = 3" - ], - "metadata": { - "collapsed": false, - "ExecuteTime": { - "end_time": "2023-11-09T11:24:20.678786Z", - "start_time": "2023-11-09T11:24:20.085220Z" - } - }, - "id": "4e5025e1dfa0343f" + ] }, { "attachments": {}, @@ -251,31 +246,31 @@ }, { "cell_type": "markdown", - "source": [ - "### Train-test split" - ], + "id": "6d7d19a7ccedf125", "metadata": { "collapsed": false }, - "id": "6d7d19a7ccedf125" + "source": [ + "### Train-test split" + ] }, { "cell_type": "code", - "execution_count": 37, - "outputs": [], - "source": [ - "Y = data_filtered[TARGET_COLUMN]\n", - "X = data_filtered.drop(columns=[TARGET_COLUMN])[list(COLUMN_TYPES.keys())]\n", - "X_train, X_test, Y_train, Y_test = model_selection.train_test_split(X, Y, random_state=RANDOM_STATE, stratify=Y)" - ], + "execution_count": 6, + "id": "1019a31f5c38dc95", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T11:24:32.909461Z", "start_time": "2023-11-09T11:24:32.904935Z" - } + }, + "collapsed": false }, - "id": "1019a31f5c38dc95" + "outputs": [], + "source": [ + "Y = data_filtered[TARGET_COLUMN]\n", + "X = data_filtered.drop(columns=[TARGET_COLUMN])[list(COLUMN_TYPES.keys())]\n", + "X_train, X_test, Y_train, Y_test = model_selection.train_test_split(X, Y, random_state=RANDOM_STATE, stratify=Y)" + ] }, { "attachments": {}, @@ -289,7 +284,7 @@ }, { "cell_type": "code", - "execution_count": 38, + "execution_count": null, "id": "b4adbbbd", "metadata": { "ExecuteTime": { @@ -385,13 +380,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], + "id": "6c5788eb97dd7e2b", "metadata": { "collapsed": false }, - "id": "6c5788eb97dd7e2b" + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -417,7 +412,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 11, "id": "ecb49fa5", "metadata": { "ExecuteTime": { @@ -428,7 +423,2709 @@ "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -440,13 +3137,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Generate comprehensive test suites automatically for your model" - ], + "id": "5651210c188e81ca", "metadata": { "collapsed": false }, - "id": "5651210c188e81ca" + "source": [ + "## Generate comprehensive test suites automatically for your model" + ] }, { "cell_type": "markdown", @@ -460,7 +3157,7 @@ }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 12, "id": "bea736a9", "metadata": { "ExecuteTime": { @@ -473,94 +3170,1018 @@ "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 11:56:58,129 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:56:58,130 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (213, 2) executed in 0:00:00.018116\n", + "2024-05-29 11:57:01,545 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:01,546 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (213, 2) executed in 0:00:00.022507\n", + "2024-05-29 11:57:01,552 pid:53166 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:03.448961\n", + "2024-05-29 11:57:01,554 pid:53166 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.000496\n", + "Executed 'Invariance to “Transform numbers to words”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", " Test failed\n", - " Metric: 0.95\n", - " - [TestMessageLevel.INFO] 213 rows were perturbed\n", + " Metric: 0.92\n", + " - [INFO] 192 rows were perturbed\n", " \n", - "Executed 'Precision on data slice “`Content` contains \"gives\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:01,567 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:01,568 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (213, 2) executed in 0:00:00.008830\n", + "2024-05-29 11:57:03,985 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,121 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (213, 2) executed in 0:00:00.155254\n", + "2024-05-29 11:57:04,123 pid:53166 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:02.564476\n", + "2024-05-29 11:57:04,124 pid:53166 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.000349\n", + "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + " Test failed\n", + " Metric: 0.93\n", + " - [INFO] 213 rows were perturbed\n", + " \n", + "2024-05-29 11:57:04,161 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,162 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (21, 2) executed in 0:00:00.017651\n", + "Executed 'Precision on data slice “`Content` contains \"gives\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.29\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"delay\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,199 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,199 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (20, 2) executed in 0:00:00.017908\n", + "Executed 'Precision on data slice “`Content` contains \"delay\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.3\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"sacramento\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,232 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,233 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.018628\n", + "Executed 'Precision on data slice “`Content` contains \"sacramento\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.3\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"dasovich\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,272 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,272 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.008717\n", + "Executed 'Precision on data slice “`Content` contains \"dasovich\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.3\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"pro\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,312 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,313 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (21, 2) executed in 0:00:00.016500\n", + "Executed 'Precision on data slice “`Content` contains \"pro\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.33\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"jeff\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,350 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,352 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (53, 2) executed in 0:00:00.017631\n", + "Executed 'Precision on data slice “`Content` contains \"jeff\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.34\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"alan\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,390 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,391 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (28, 2) executed in 0:00:00.013602\n", + "Executed 'Precision on data slice “`Content` contains \"alan\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.36\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"judge\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,426 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,427 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (21, 2) executed in 0:00:00.017925\n", + "Executed 'Precision on data slice “`Content` contains \"judge\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.38\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"blackouts\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,458 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,459 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (21, 2) executed in 0:00:00.019073\n", + "Executed 'Precision on data slice “`Content` contains \"blackouts\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.38\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"emergency\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,498 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,499 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.017635\n", + "Executed 'Precision on data slice “`Content` contains \"push\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.39\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"push\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,535 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,536 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.019982\n", + "Executed 'Precision on data slice “`Content` contains \"emergency\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.39\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"fair\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,568 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,568 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (35, 2) executed in 0:00:00.019893\n", + "Executed 'Precision on data slice “`Content` contains \"governor\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"duke\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,609 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,609 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (25, 2) executed in 0:00:00.018383\n", + "Executed 'Precision on data slice “`Content` contains \"duke\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"governor\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,648 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,649 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (20, 2) executed in 0:00:00.018559\n", + "Executed 'Precision on data slice “`Content` contains \"fair\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`Content` contains \"friday\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", + "2024-05-29 11:57:04,686 pid:53166 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Content': 'object'} to {'Content': 'object'}\n", + "2024-05-29 11:57:04,687 pid:53166 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (42, 2) executed in 0:00:00.020775\n", + "Executed 'Precision on data slice “`Content` contains \"friday\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}: \n", " Test failed\n", " Metric: 0.4\n", " \n", - " \n" + " \n", + "2024-05-29 11:57:04,689 pid:53166 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 11:57:04,689 pid:53166 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 11:57:04,690 pid:53166 MainThread giskard.core.suite INFO Invariance to “Transform numbers to words” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.921875}\n", + "2024-05-29 11:57:04,690 pid:53166 MainThread giskard.core.suite INFO Invariance to “Add typos” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.9342723004694836}\n", + "2024-05-29 11:57:04,690 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"gives\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.2857142857142857}\n", + "2024-05-29 11:57:04,690 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"delay\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.3}\n", + "2024-05-29 11:57:04,691 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"sacramento\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.30434782608695654}\n", + "2024-05-29 11:57:04,691 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"dasovich\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.30434782608695654}\n", + "2024-05-29 11:57:04,691 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"pro\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.3333333333333333}\n", + "2024-05-29 11:57:04,691 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"jeff\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.33962264150943394}\n", + "2024-05-29 11:57:04,692 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"alan\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.35714285714285715}\n", + "2024-05-29 11:57:04,692 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"judge\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.38095238095238093}\n", + "2024-05-29 11:57:04,692 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"blackouts\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.38095238095238093}\n", + "2024-05-29 11:57:04,693 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"push\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.391304347826087}\n", + "2024-05-29 11:57:04,693 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"emergency\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.391304347826087}\n", + "2024-05-29 11:57:04,694 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"governor\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.4}\n", + "2024-05-29 11:57:04,694 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"duke\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.4}\n", + "2024-05-29 11:57:04,695 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"fair\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.4}\n", + "2024-05-29 11:57:04,695 pid:53166 MainThread giskard.core.suite INFO Precision on data slice “`Content` contains \"friday\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.544131455399061}): {failed, metric=0.40476190476190477}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Invariance to “Add typos”\n
\n \n Measured Metric = 0.94836\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n transformation_function\n Add typos\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`Content` contains "gives"”\n
\n \n Measured Metric = 0.28571\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "gives"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`Content` contains "delay"”\n
\n \n Measured Metric = 0.3\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "delay"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`Content` contains "sacramento"”\n
\n \n Measured Metric = 0.30435\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "sacramento"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "dasovich"”\n
\n \n Measured Metric = 0.30435\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "dasovich"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "pro"”\n
\n \n Measured Metric = 0.33333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "pro"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "jeff"”\n
\n \n Measured Metric = 0.33962\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "jeff"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "alan"”\n
\n \n Measured Metric = 0.35714\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "alan"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "judge"”\n
\n \n Measured Metric = 0.38095\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "judge"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "blackouts"”\n
\n \n Measured Metric = 0.38095\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "blackouts"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "emergency"”\n
\n \n Measured Metric = 0.3913\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "emergency"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "push"”\n
\n \n Measured Metric = 0.3913\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "push"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "fair"”\n
\n \n Measured Metric = 0.4\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "fair"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "duke"”\n
\n \n Measured Metric = 0.4\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "duke"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "governor"”\n
\n \n Measured Metric = 0.4\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "governor"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Content` contains "friday"”\n
\n \n Measured Metric = 0.40476\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 3ef6293a-8420-40ee-bb44-57602b025a6f\n
\n \n
\n dataset\n Email classifier\n
\n \n
\n slicing_function\n `Content` contains "friday"\n
\n \n
\n threshold\n 0.544131455399061\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Transform numbers to words”\n", + "
\n", + " \n", + " Measured Metric = 0.92188\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Transform numbers to words\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Add typos”\n", + "
\n", + " \n", + " Measured Metric = 0.93427\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Add typos\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "gives"”\n", + "
\n", + " \n", + " Measured Metric = 0.28571\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "gives"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "delay"”\n", + "
\n", + " \n", + " Measured Metric = 0.3\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "delay"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "sacramento"”\n", + "
\n", + " \n", + " Measured Metric = 0.30435\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "sacramento"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "dasovich"”\n", + "
\n", + " \n", + " Measured Metric = 0.30435\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "dasovich"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "pro"”\n", + "
\n", + " \n", + " Measured Metric = 0.33333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "pro"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "jeff"”\n", + "
\n", + " \n", + " Measured Metric = 0.33962\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "jeff"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "alan"”\n", + "
\n", + " \n", + " Measured Metric = 0.35714\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "alan"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "judge"”\n", + "
\n", + " \n", + " Measured Metric = 0.38095\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "judge"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "blackouts"”\n", + "
\n", + " \n", + " Measured Metric = 0.38095\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "blackouts"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "push"”\n", + "
\n", + " \n", + " Measured Metric = 0.3913\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "push"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "emergency"”\n", + "
\n", + " \n", + " Measured Metric = 0.3913\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "emergency"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "governor"”\n", + "
\n", + " \n", + " Measured Metric = 0.4\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "governor"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "duke"”\n", + "
\n", + " \n", + " Measured Metric = 0.4\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "duke"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "fair"”\n", + "
\n", + " \n", + " Measured Metric = 0.4\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "fair"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Content` contains "friday"”\n", + "
\n", + " \n", + " Measured Metric = 0.40476\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Email category classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Emails of different categories\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Content` contains "friday"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.544131455399061\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 45, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -591,132 +4212,14 @@ { "cell_type": "code", "execution_count": null, - "outputs": [], - "source": [ - "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" - ], - "metadata": { - "collapsed": false - }, - "id": "13c049a762dcc4f" - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - }, - "id": "42bef6e343b97544" - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], + "id": "13c049a762dcc4f", "metadata": { "collapsed": false }, - "id": "36f7ffad0a45f60e" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - }, - "id": "8f25937d7f7f138e" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - }, - "id": "cfe814d192e70ca1" - }, - { - "cell_type": "markdown", - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ], - "metadata": { - "collapsed": false - }, - "id": "f4e667202aed8fbd" - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8efd6bf3", - "metadata": {}, "outputs": [], "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "id": "7f594b5a762b09", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" + "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - }, - "id": "c84fad7b733e0553" } ], "metadata": { @@ -735,7 +4238,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.11" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/fake_real_news_classification.ipynb b/docs/reference/notebooks/fake_real_news_classification.ipynb index 40e9fa9766..cb74b26e6b 100644 --- a/docs/reference/notebooks/fake_real_news_classification.ipynb +++ b/docs/reference/notebooks/fake_real_news_classification.ipynb @@ -19,12 +19,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -61,7 +56,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": { "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0", "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a", @@ -86,32 +81,32 @@ "from sklearn.model_selection import train_test_split\n", "from typing import Tuple, Callable\n", "\n", - "from giskard import Dataset, Model, scan, testing, GiskardClient, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { "cell_type": "markdown", - "source": [ - "## Notebook-level settings" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Notebook-level settings" + ] }, { "cell_type": "code", "execution_count": 2, - "outputs": [], - "source": [ - "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'" - ], "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:28:18.418915Z", "start_time": "2023-11-09T12:28:18.408860Z" - } - } + }, + "collapsed": false + }, + "outputs": [], + "source": [ + "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'" + ] }, { "cell_type": "markdown", @@ -126,11 +121,11 @@ "cell_type": "code", "execution_count": 3, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:28:21.279141Z", "start_time": "2023-11-09T12:28:21.238638Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -226,11 +221,11 @@ "cell_type": "code", "execution_count": 5, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:28:24.341850Z", "start_time": "2023-11-09T12:28:24.290063Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -257,13 +252,13 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:28:25.530437Z", "start_time": "2023-11-09T12:28:25.487051Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -298,11 +293,11 @@ "cell_type": "code", "execution_count": 7, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:28:32.029185Z", "start_time": "2023-11-09T12:28:27.412721Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -357,12 +352,12 @@ "cell_type": "code", "execution_count": 8, "metadata": { - "_kg_hide-output": true, - "trusted": true, "ExecuteTime": { "end_time": "2023-11-09T12:29:05.258459Z", "start_time": "2023-11-09T12:28:32.775183Z" - } + }, + "_kg_hide-output": true, + "trusted": true }, "outputs": [], "source": [ @@ -509,14 +504,14 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Scan your model for vulnerabilities with Giskard\n", "\n", "Giskard's scan allows you to detect vulnerabilities in your model automatically. These include performance biases, unrobustness, data leakage, stochasticity, underconfidence, ethical issues, and more. For detailed information about the scan feature, please refer to our [scan documentation](https://docs.giskard.ai/en/stable/open_source/scan/scan_nlp/index.html)." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", @@ -533,16 +528,1327 @@ "cell_type": "code", "execution_count": 12, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:41:37.561048Z", "start_time": "2023-11-09T12:41:37.297675Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -563,106 +1869,461 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", "execution_count": 13, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:42:37.991107Z", "start_time": "2023-11-09T12:42:36.923834Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Recall on data slice “`text` contains \"october\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9306910569105691}: \n", - " Test failed\n", - " Metric: 0.87\n", - " \n", - " \n", - "Executed 'Accuracy on data slice “`avg_whitespace(title)` >= 0.147 AND `avg_whitespace(title)` < 0.152”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9386}: \n", + "2024-05-29 12:04:25,717 pid:57157 MainThread giskard.datasets.base INFO Casting dataframe columns from {'title': 'object', 'text': 'object'} to {'title': 'object', 'text': 'object'}\n", + "2024-05-29 12:04:25,720 pid:57157 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 3) executed in 0:00:00.013458\n", + "Executed 'Recall on data slice “`text_length(text)` >= 2570.500 AND `text_length(text)` < 2729.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}: \n", " Test failed\n", " Metric: 0.91\n", " \n", " \n", - "Executed 'Recall on data slice “`text_length(title)` >= 92.500 AND `text_length(title)` < 97.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9306910569105691}: \n", - " Test failed\n", - " Metric: 0.9\n", - " \n", - " \n", - "Executed 'Recall on data slice “`text` contains \"decision\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9306910569105691}: \n", + "2024-05-29 12:04:25,756 pid:57157 MainThread giskard.datasets.base INFO Casting dataframe columns from {'title': 'object', 'text': 'object'} to {'title': 'object', 'text': 'object'}\n", + "2024-05-29 12:04:25,757 pid:57157 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (42, 3) executed in 0:00:00.006029\n", + "Executed 'Recall on data slice “`text` contains \"texas\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}: \n", " Test failed\n", " Metric: 0.91\n", " \n", " \n", - "Executed 'Recall on data slice “`text` contains \"texas\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9306910569105691}: \n", + "2024-05-29 12:04:25,781 pid:57157 MainThread giskard.datasets.base INFO Casting dataframe columns from {'title': 'object', 'text': 'object'} to {'title': 'object', 'text': 'object'}\n", + "2024-05-29 12:04:25,782 pid:57157 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (38, 3) executed in 0:00:00.005618\n", + "Executed 'Recall on data slice “`text` contains \"october\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}: \n", " Test failed\n", " Metric: 0.91\n", " \n", " \n", - "Executed 'Recall on data slice “`text` contains \"life\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9306910569105691}: \n", - " Test failed\n", - " Metric: 0.92\n", - " \n", - " \n", - "Executed 'Accuracy on data slice “`text` contains \"guilty\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9386}: \n", - " Test failed\n", - " Metric: 0.93\n", - " \n", - " \n", - "Executed 'Recall on data slice “`text` contains \"september\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9306910569105691}: \n", - " Test failed\n", - " Metric: 0.92\n", - " \n", - " \n", - "Executed 'Accuracy on data slice “`avg_word_length(text)` < 4.884 AND `avg_word_length(text)` >= 4.835”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9386}: \n", - " Test failed\n", - " Metric: 0.93\n", - " \n", - " \n", - "Executed 'Accuracy on data slice “`avg_digits(text)` >= 0.007 AND `avg_digits(text)` < 0.008”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9386}: \n", + "2024-05-29 12:04:25,808 pid:57157 MainThread giskard.datasets.base INFO Casting dataframe columns from {'title': 'object', 'text': 'object'} to {'title': 'object', 'text': 'object'}\n", + "2024-05-29 12:04:25,808 pid:57157 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (27, 3) executed in 0:00:00.005583\n", + "Executed 'Accuracy on data slice “`text` contains \"guilty\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9424}: \n", " Test failed\n", " Metric: 0.93\n", " \n", " \n", - "Executed 'Accuracy on data slice “`text_length(text)` >= 2446.500 AND `text_length(text)` < 2572.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9386}: \n", + "2024-05-29 12:04:25,832 pid:57157 MainThread giskard.datasets.base INFO Casting dataframe columns from {'title': 'object', 'text': 'object'} to {'title': 'object', 'text': 'object'}\n", + "2024-05-29 12:04:25,833 pid:57157 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (61, 3) executed in 0:00:00.006272\n", + "Executed 'Recall on data slice “`text` contains \"investigation\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}: \n", " Test failed\n", " Metric: 0.93\n", " \n", " \n", - "Executed 'Accuracy on data slice “`title` contains \"video\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9386}: \n", - " Test failed\n", - " Metric: 0.94\n", - " \n", - " \n", - "Executed 'Recall on data slice “`avg_digits(text)` >= 0.003 AND `avg_digits(text)` < 0.004”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9306910569105691}: \n", + "2024-05-29 12:04:25,844 pid:57157 MainThread giskard.datasets.base INFO Casting dataframe columns from {'title': 'object', 'text': 'object'} to {'title': 'object', 'text': 'object'}\n", + "2024-05-29 12:04:25,845 pid:57157 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (33, 3) executed in 0:00:00.006265\n", + "Executed 'Recall on data slice “`text_length(title)` >= 92.500 AND `text_length(title)` < 97.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}: \n", " Test failed\n", " Metric: 0.93\n", " \n", " \n", - "Executed 'Accuracy on data slice “`text` contains \"elections\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9386}: \n", - " Test failed\n", - " Metric: 0.94\n", - " \n", - " \n" + "2024-05-29 12:04:25,847 pid:57157 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 12:04:25,847 pid:57157 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 12:04:25,848 pid:57157 MainThread giskard.core.suite INFO Recall on data slice “`text_length(text)` >= 2570.500 AND `text_length(text)` < 2729.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}): {failed, metric=0.9090909090909091}\n", + "2024-05-29 12:04:25,848 pid:57157 MainThread giskard.core.suite INFO Recall on data slice “`text` contains \"texas\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}): {failed, metric=0.9090909090909091}\n", + "2024-05-29 12:04:25,849 pid:57157 MainThread giskard.core.suite INFO Recall on data slice “`text` contains \"october\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}): {failed, metric=0.9130434782608695}\n", + "2024-05-29 12:04:25,849 pid:57157 MainThread giskard.core.suite INFO Accuracy on data slice “`text` contains \"guilty\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9424}): {failed, metric=0.9259259259259259}\n", + "2024-05-29 12:04:25,850 pid:57157 MainThread giskard.core.suite INFO Recall on data slice “`text` contains \"investigation\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}): {failed, metric=0.9333333333333333}\n", + "2024-05-29 12:04:25,850 pid:57157 MainThread giskard.core.suite INFO Recall on data slice “`text_length(title)` >= 92.500 AND `text_length(title)` < 97.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.9384146341463414}): {failed, metric=0.9333333333333333}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Recall on data slice “`text` contains "october"”\n
\n \n Measured Metric = 0.86957\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text` contains "october"\n
\n \n
\n threshold\n 0.9306910569105691\n
\n \n
\n
\n \n \n
\n Test Accuracy on data slice “`avg_whitespace(title)` >= 0.147 AND `avg_whitespace(title)` < 0.152”\n
\n \n Measured Metric = 0.90625\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `avg_whitespace(title)` >= 0.147 AND `avg_whitespace(title)` < 0.152\n
\n \n
\n threshold\n 0.9386\n
\n \n
\n
\n \n \n
\n Test Recall on data slice “`text_length(title)` >= 92.500 AND `text_length(title)` < 97.500”\n
\n \n Measured Metric = 0.9\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text_length(title)` >= 92.500 AND `text_length(title)` < 97.500\n
\n \n
\n threshold\n 0.9306910569105691\n
\n \n
\n
\n \n \n
\n Test Recall on data slice “`text` contains "decision"”\n
\n \n Measured Metric = 0.90909\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text` contains "decision"\n
\n \n
\n threshold\n 0.9306910569105691\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`text` contains "texas"”\n
\n \n Measured Metric = 0.90909\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text` contains "texas"\n
\n \n
\n threshold\n 0.9306910569105691\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`text` contains "life"”\n
\n \n Measured Metric = 0.91667\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text` contains "life"\n
\n \n
\n threshold\n 0.9306910569105691\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`text` contains "guilty"”\n
\n \n Measured Metric = 0.92593\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text` contains "guilty"\n
\n \n
\n threshold\n 0.9386\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`text` contains "september"”\n
\n \n Measured Metric = 0.92\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text` contains "september"\n
\n \n
\n threshold\n 0.9306910569105691\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`avg_word_length(text)` < 4.884 AND `avg_word_length(text)` >= 4.835”\n
\n \n Measured Metric = 0.93333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `avg_word_length(text)` < 4.884 AND `avg_word_length(text)` >= 4.835\n
\n \n
\n threshold\n 0.9386\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`avg_digits(text)` >= 0.007 AND `avg_digits(text)` < 0.008”\n
\n \n Measured Metric = 0.93333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `avg_digits(text)` >= 0.007 AND `avg_digits(text)` < 0.008\n
\n \n
\n threshold\n 0.9386\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`text_length(text)` >= 2446.500 AND `text_length(text)` < 2572.500”\n
\n \n Measured Metric = 0.93333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text_length(text)` >= 2446.500 AND `text_length(text)` < 2572.500\n
\n \n
\n threshold\n 0.9386\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`title` contains "video"”\n
\n \n Measured Metric = 0.93548\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `title` contains "video"\n
\n \n
\n threshold\n 0.9386\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`avg_digits(text)` >= 0.003 AND `avg_digits(text)` < 0.004”\n
\n \n Measured Metric = 0.92857\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `avg_digits(text)` >= 0.003 AND `avg_digits(text)` < 0.004\n
\n \n
\n threshold\n 0.9306910569105691\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`text` contains "elections"”\n
\n \n Measured Metric = 0.9375\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5135f9ad-6d07-4db6-af80-a84dd1965396\n
\n \n
\n dataset\n fake_and_real_news\n
\n \n
\n slicing_function\n `text` contains "elections"\n
\n \n
\n threshold\n 0.9386\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`text_length(text)` >= 2570.500 AND `text_length(text)` < 2729.500”\n", + "
\n", + " \n", + " Measured Metric = 0.90909\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " fake_real_news_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fake_and_real_news\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text_length(text)` >= 2570.500 AND `text_length(text)` < 2729.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9384146341463414\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`text` contains "texas"”\n", + "
\n", + " \n", + " Measured Metric = 0.90909\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " fake_real_news_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fake_and_real_news\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text` contains "texas"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9384146341463414\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`text` contains "october"”\n", + "
\n", + " \n", + " Measured Metric = 0.91304\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " fake_real_news_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fake_and_real_news\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text` contains "october"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9384146341463414\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`text` contains "guilty"”\n", + "
\n", + " \n", + " Measured Metric = 0.92593\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " fake_real_news_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fake_and_real_news\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text` contains "guilty"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9424\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`text` contains "investigation"”\n", + "
\n", + " \n", + " Measured Metric = 0.93333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " fake_real_news_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fake_and_real_news\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text` contains "investigation"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9384146341463414\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`text_length(title)` >= 92.500 AND `text_length(title)` < 97.500”\n", + "
\n", + " \n", + " Measured Metric = 0.93333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " fake_real_news_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fake_and_real_news\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text_length(title)` >= 92.500 AND `text_length(title)` < 97.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.9384146341463414\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 13, "metadata": {}, @@ -703,118 +2364,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces.\n" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - " \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -833,7 +2382,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.6" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/hotel_text_regression.ipynb b/docs/reference/notebooks/hotel_text_regression.ipynb index 4471b6e6ab..74d4538f7a 100644 --- a/docs/reference/notebooks/hotel_text_regression.ipynb +++ b/docs/reference/notebooks/hotel_text_regression.ipynb @@ -21,12 +21,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -63,11 +58,11 @@ "cell_type": "code", "execution_count": 1, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:11:53.867789Z", "start_time": "2023-11-09T12:11:49.171837Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -83,7 +78,7 @@ "from sklearn.preprocessing import FunctionTransformer\n", "from typing import Iterable\n", "\n", - "from giskard import Model, Dataset, scan, testing, GiskardClient, Suite" + "from giskard import Model, Dataset, scan, testing" ] }, { @@ -99,11 +94,11 @@ "cell_type": "code", "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:12:05.303464Z", "start_time": "2023-11-09T12:12:05.254149Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -139,11 +134,11 @@ "cell_type": "code", "execution_count": 3, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:12:21.189537Z", "start_time": "2023-11-09T12:12:21.170978Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -194,12 +189,12 @@ "cell_type": "code", "execution_count": 5, "metadata": { - "_cell_guid": "1fc3041b-4143-4913-be91-522a80491717", - "_uuid": "6edbd3a2e85aced1897d44dbabf74ebfecf10110", "ExecuteTime": { "end_time": "2023-11-09T12:12:40.137753Z", "start_time": "2023-11-09T12:12:40.084154Z" - } + }, + "_cell_guid": "1fc3041b-4143-4913-be91-522a80491717", + "_uuid": "6edbd3a2e85aced1897d44dbabf74ebfecf10110" }, "outputs": [], "source": [ @@ -219,13 +214,13 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:12:51.323768Z", "start_time": "2023-11-09T12:12:51.270599Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -260,11 +255,11 @@ "cell_type": "code", "execution_count": 7, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:12:53.290346Z", "start_time": "2023-11-09T12:12:53.254228Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -336,7 +331,7 @@ }, "outputs": [], "source": [ - "# Wrap the prediction method to allow saving the whole pipeline to the Hub\n", + "# Wrap the prediction method\n", "def prediction_function(df):\n", " return pipeline.predict(df)\n", "\n", @@ -357,12 +352,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -390,16 +385,1773 @@ "cell_type": "code", "execution_count": 11, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:13:26.545786Z", "start_time": "2023-11-09T12:13:26.215236Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -420,81 +2172,623 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", "execution_count": 12, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:13:35.489958Z", "start_time": "2023-11-09T12:13:29.988997Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 17:25:08,294 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,295 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (250, 2) executed in 0:00:00.003143\n", + "2024-05-29 17:25:08,317 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,322 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (250, 2) executed in 0:00:00.008373\n", + "2024-05-29 17:25:08,325 pid:11998 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:00.781405\n", + "2024-05-29 17:25:08,326 pid:11998 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.001536\n", + "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", " Test failed\n", - " Metric: 0.87\n", - " - [TestMessageLevel.INFO] 239 rows were perturbed\n", + " Metric: 0.91\n", + " - [INFO] 241 rows were perturbed\n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"building\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,335 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,337 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (25, 2) executed in 0:00:00.004939\n", + "Executed 'MSE on data slice “`Full_Review` contains \"building\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 3.6\n", " \n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"stay\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,345 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,347 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (25, 2) executed in 0:00:00.004945\n", + "Executed 'MSE on data slice “`Full_Review` contains \"stay\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 3.4\n", " \n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"bed\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,354 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,356 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (46, 2) executed in 0:00:00.003780\n", + "Executed 'MSE on data slice “`Full_Review` contains \"bed\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 2.96\n", " \n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"comfy\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,364 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,365 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.005052\n", + "Executed 'MSE on data slice “`Full_Review` contains \"comfy\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 2.72\n", " \n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"area\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,374 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,375 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (24, 2) executed in 0:00:00.004607\n", + "Executed 'MSE on data slice “`Full_Review` contains \"area\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 2.63\n", " \n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"food\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,385 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,386 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.005924\n", + "Executed 'MSE on data slice “`Full_Review` contains \"food\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 2.57\n", " \n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"hotel\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,394 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,396 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (80, 2) executed in 0:00:00.005724\n", + "Executed 'MSE on data slice “`Full_Review` contains \"hotel\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 2.52\n", " \n", " \n", - "Executed 'MSE on data slice “`Full_Review` contains \"room\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", + "2024-05-29 17:25:08,406 pid:11998 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Full_Review': 'object'} to {'Full_Review': 'object'}\n", + "2024-05-29 17:25:08,406 pid:11998 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (124, 2) executed in 0:00:00.003998\n", + "Executed 'MSE on data slice “`Full_Review` contains \"room\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}: \n", " Test failed\n", " Metric: 2.4\n", " \n", - " \n" + " \n", + "2024-05-29 17:25:08,408 pid:11998 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 17:25:08,409 pid:11998 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 17:25:08,409 pid:11998 MainThread giskard.core.suite INFO Invariance to “Add typos” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.9087136929460581}\n", + "2024-05-29 17:25:08,409 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"building\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=3.600595896699133}\n", + "2024-05-29 17:25:08,410 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"stay\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=3.3984339534286825}\n", + "2024-05-29 17:25:08,410 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"bed\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=2.9552583451970587}\n", + "2024-05-29 17:25:08,410 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"comfy\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=2.7172616774396765}\n", + "2024-05-29 17:25:08,411 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"area\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=2.6276261964130216}\n", + "2024-05-29 17:25:08,411 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"food\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=2.57312044778542}\n", + "2024-05-29 17:25:08,411 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"hotel\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=2.515134565551532}\n", + "2024-05-29 17:25:08,412 pid:11998 MainThread giskard.core.suite INFO MSE on data slice “`Full_Review` contains \"room\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 2.3372302706019896}): {failed, metric=2.4023965763964417}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Invariance to “Add typos”\n
\n \n Measured Metric = 0.87448\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n transformation_function\n Add typos\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`Full_Review` contains "building"”\n
\n \n Measured Metric = 3.6006\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "building"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`Full_Review` contains "stay"”\n
\n \n Measured Metric = 3.39843\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "stay"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`Full_Review` contains "bed"”\n
\n \n Measured Metric = 2.95526\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "bed"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n \n \n \n
\n Test MSE on data slice “`Full_Review` contains "comfy"”\n
\n \n Measured Metric = 2.71726\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "comfy"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n \n \n \n
\n Test MSE on data slice “`Full_Review` contains "area"”\n
\n \n Measured Metric = 2.62763\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "area"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n \n \n \n
\n Test MSE on data slice “`Full_Review` contains "food"”\n
\n \n Measured Metric = 2.57312\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "food"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n \n \n \n
\n Test MSE on data slice “`Full_Review` contains "hotel"”\n
\n \n Measured Metric = 2.51513\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "hotel"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n \n \n \n
\n Test MSE on data slice “`Full_Review` contains "room"”\n
\n \n Measured Metric = 2.4024\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 5fb285f3-16dc-4a3f-8be6-0b495d9a5e41\n
\n \n
\n dataset\n hotel_text_regression_dataset\n
\n \n
\n slicing_function\n `Full_Review` contains "room"\n
\n \n
\n threshold\n 2.3372302706019896\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Add typos”\n", + "
\n", + " \n", + " Measured Metric = 0.90871\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Add typos\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "building"”\n", + "
\n", + " \n", + " Measured Metric = 3.6006\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "building"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "stay"”\n", + "
\n", + " \n", + " Measured Metric = 3.39843\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "stay"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "bed"”\n", + "
\n", + " \n", + " Measured Metric = 2.95526\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "bed"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "comfy"”\n", + "
\n", + " \n", + " Measured Metric = 2.71726\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "comfy"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "area"”\n", + "
\n", + " \n", + " Measured Metric = 2.62763\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "area"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "food"”\n", + "
\n", + " \n", + " Measured Metric = 2.57312\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "food"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "hotel"”\n", + "
\n", + " \n", + " Measured Metric = 2.51513\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "hotel"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`Full_Review` contains "room"”\n", + "
\n", + " \n", + " Measured Metric = 2.4024\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " hotel_text_regression\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " hotel_text_regression_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Full_Review` contains "room"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 2.3372302706019896\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 12, "metadata": {}, @@ -535,118 +2829,6 @@ "source": [ "test_suite.add_test(testing.test_r2(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -665,7 +2847,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.1" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/ieee_fraud_detection_adversarial_validation.ipynb b/docs/reference/notebooks/ieee_fraud_detection_adversarial_validation.ipynb index d8352fd5ee..3e98100999 100644 --- a/docs/reference/notebooks/ieee_fraud_detection_adversarial_validation.ipynb +++ b/docs/reference/notebooks/ieee_fraud_detection_adversarial_validation.ipynb @@ -22,12 +22,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -75,11 +70,11 @@ "cell_type": "code", "execution_count": 1, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:17:43.454764Z", "start_time": "2023-11-09T12:17:37.151513Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -94,7 +89,7 @@ "from sklearn.metrics import roc_auc_score\n", "from sklearn.model_selection import train_test_split\n", "\n", - "from giskard import GiskardClient, scan, testing, Dataset, Model, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { @@ -110,11 +105,11 @@ "cell_type": "code", "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:17:44.751420Z", "start_time": "2023-11-09T12:17:44.719440Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -149,11 +144,11 @@ "cell_type": "code", "execution_count": 3, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:17:45.925766Z", "start_time": "2023-11-09T12:17:45.904823Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -256,11 +251,11 @@ "cell_type": "code", "execution_count": 4, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:17:46.316557Z", "start_time": "2023-11-09T12:17:46.290804Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -320,11 +315,11 @@ "cell_type": "code", "execution_count": 6, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:18:39.711301Z", "start_time": "2023-11-09T12:18:39.634839Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -344,13 +339,13 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:18:40.628414Z", "start_time": "2023-11-09T12:18:40.394073Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -431,11 +426,11 @@ "cell_type": "code", "execution_count": 9, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:18:42.538241Z", "start_time": "2023-11-09T12:18:42.501449Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -467,12 +462,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -500,16 +495,2509 @@ "cell_type": "code", "execution_count": 12, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:25:58.669132Z", "start_time": "2023-11-09T12:25:58.390530Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -530,111 +3018,920 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", "execution_count": 13, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:26:06.901452Z", "start_time": "2023-11-09T12:26:00.832299Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Precision on data slice “`D15` >= 4.000 AND `D15` < 344.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:04,603 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:04,624 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (34, 434) executed in 0:00:00.050286\n", + "Executed 'Accuracy on data slice “`D3` >= 13.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.7\n", + " Metric: 0.68\n", " \n", " \n", - "Executed 'Precision on data slice “`D4` >= 81.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:04,676 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:04,696 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (31, 434) executed in 0:00:00.042018\n", + "Executed 'Accuracy on data slice “`D5` >= 10.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.75\n", + " Metric: 0.68\n", " \n", " \n", - "Executed 'Accuracy on data slice “`D2` >= 108.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8664000000000001}: \n", + "2024-05-29 13:32:04,751 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:04,771 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (37, 434) executed in 0:00:00.044460\n", + "Executed 'Accuracy on data slice “`D11` >= 31.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.79\n", + " Metric: 0.7\n", " \n", " \n", - "Executed 'Accuracy on data slice “`C6` >= 1.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8664000000000001}: \n", + "2024-05-29 13:32:04,826 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:04,847 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (31, 434) executed in 0:00:00.046466\n", + "Executed 'Accuracy on data slice “`card1` < 12740.000 AND `card1` >= 7867.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.8\n", + " Metric: 0.71\n", " \n", " \n", - "Executed 'Accuracy on data slice “`D11` >= 65.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8664000000000001}: \n", + "2024-05-29 13:32:04,982 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,004 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (32, 434) executed in 0:00:00.126585\n", + "Executed 'Accuracy on data slice “`M8` == \"F\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.81\n", + " Metric: 0.72\n", " \n", " \n", - "Executed 'Precision on data slice “`V310` >= 48.475”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:05,062 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,087 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (38, 434) executed in 0:00:00.051907\n", + "Executed 'Precision on data slice “`D15` >= 1.000 AND `D15` < 232.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}: \n", " Test failed\n", - " Metric: 0.79\n", + " Metric: 0.68\n", " \n", " \n", - "Executed 'Precision on data slice “`C11` >= 1.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:05,144 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,166 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (42, 434) executed in 0:00:00.046181\n", + "Executed 'Accuracy on data slice “`M7` == \"F\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.79\n", + " Metric: 0.74\n", " \n", " \n", - "Executed 'Accuracy on data slice “`D5` >= 13.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8664000000000001}: \n", + "2024-05-29 13:32:05,221 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,244 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (31, 434) executed in 0:00:00.046322\n", + "Executed 'Accuracy on data slice “`D1` >= 0.500 AND `D1` < 120.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.81\n", + " Metric: 0.74\n", " \n", " \n", - "Executed 'Precision on data slice “`C1` >= 2.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:05,301 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,324 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (50, 434) executed in 0:00:00.049778\n", + "Executed 'Precision on data slice “`C9` >= 0.500 AND `C9` < 1.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}: \n", " Test failed\n", - " Metric: 0.8\n", + " Metric: 0.69\n", " \n", " \n", - "Executed 'Precision on data slice “`V283` < 0.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:05,505 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,525 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (35, 434) executed in 0:00:00.042524\n", + "Executed 'Accuracy on data slice “`V10` < 0.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.8\n", + " Metric: 0.77\n", " \n", " \n", - "Executed 'Precision on data slice “`V282` < 0.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:05,579 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,601 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (35, 434) executed in 0:00:00.045713\n", + "Executed 'Accuracy on data slice “`V11` < 0.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.8\n", + " Metric: 0.77\n", " \n", " \n", - "Executed 'Accuracy on data slice “`D3` >= 13.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8664000000000001}: \n", + "2024-05-29 13:32:05,656 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,678 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (49, 434) executed in 0:00:00.045881\n", + "Executed 'Precision on data slice “`P_emaildomain` == \"gmail.com\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}: \n", " Test failed\n", - " Metric: 0.82\n", + " Metric: 0.71\n", " \n", " \n", - "Executed 'Precision on data slice “`V285` >= 0.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8444444444444444}: \n", + "2024-05-29 13:32:05,731 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,753 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (46, 434) executed in 0:00:00.045060\n", + "Executed 'Precision on data slice “`M6` == \"T\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}: \n", " Test failed\n", - " Metric: 0.81\n", + " Metric: 0.71\n", " \n", " \n", - "Executed 'Recall on data slice “`addr1` >= 312.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8603773584905661}: \n", + "2024-05-29 13:32:05,809 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,909 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (40, 434) executed in 0:00:00.125281\n", + "Executed 'Accuracy on data slice “`C6` >= 1.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.83\n", + " Metric: 0.78\n", " \n", " \n", - "Executed 'Accuracy on data slice “`D10` >= 222.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8664000000000001}: \n", + "2024-05-29 13:32:05,959 pid:62817 MainThread giskard.datasets.base INFO Casting dataframe columns from {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'} to {'TransactionAmt': 'float32', 'ProductCD': 'category', 'card1': 'int16', 'card2': 'float32', 'card3': 'float32', 'card4': 'category', 'card5': 'float32', 'card6': 'category', 'addr1': 'float32', 'addr2': 'float32', 'dist1': 'float32', 'dist2': 'float32', 'P_emaildomain': 'category', 'R_emaildomain': 'category', 'C1': 'float32', 'C2': 'float32', 'C3': 'float32', 'C4': 'float32', 'C5': 'float32', 'C6': 'float32', 'C7': 'float32', 'C8': 'float32', 'C9': 'float32', 'C10': 'float32', 'C11': 'float32', 'C12': 'float32', 'C13': 'float32', 'C14': 'float32', 'D1': 'float32', 'D2': 'float32', 'D3': 'float32', 'D4': 'float32', 'D5': 'float32', 'D6': 'float32', 'D7': 'float32', 'D8': 'float32', 'D9': 'float32', 'D10': 'float32', 'D11': 'float32', 'D12': 'float32', 'D13': 'float32', 'D14': 'float32', 'D15': 'float32', 'M1': 'category', 'M2': 'category', 'M3': 'category', 'M4': 'category', 'M5': 'category', 'M6': 'category', 'M7': 'category', 'M8': 'category', 'M9': 'category', 'V1': 'float32', 'V2': 'float32', 'V3': 'float32', 'V4': 'float32', 'V5': 'float32', 'V6': 'float32', 'V7': 'float32', 'V8': 'float32', 'V9': 'float32', 'V10': 'float32', 'V11': 'float32', 'V12': 'float32', 'V13': 'float32', 'V14': 'float32', 'V15': 'float32', 'V16': 'float32', 'V17': 'float32', 'V18': 'float32', 'V19': 'float32', 'V20': 'float32', 'V21': 'float32', 'V22': 'float32', 'V23': 'float32', 'V24': 'float32', 'V25': 'float32', 'V26': 'float32', 'V27': 'float32', 'V28': 'float32', 'V29': 'float32', 'V30': 'float32', 'V31': 'float32', 'V32': 'float32', 'V33': 'float32', 'V34': 'float32', 'V35': 'float32', 'V36': 'float32', 'V37': 'float32', 'V38': 'float32', 'V39': 'float32', 'V40': 'float32', 'V41': 'float32', 'V42': 'float32', 'V43': 'float32', 'V44': 'float32', 'V45': 'float32', 'V46': 'float32', 'V47': 'float32', 'V48': 'float32', 'V49': 'float32', 'V50': 'float32', 'V51': 'float32', 'V52': 'float32', 'V53': 'float32', 'V54': 'float32', 'V55': 'float32', 'V56': 'float32', 'V57': 'float32', 'V58': 'float32', 'V59': 'float32', 'V60': 'float32', 'V61': 'float32', 'V62': 'float32', 'V63': 'float32', 'V64': 'float32', 'V65': 'float32', 'V66': 'float32', 'V67': 'float32', 'V68': 'float32', 'V69': 'float32', 'V70': 'float32', 'V71': 'float32', 'V72': 'float32', 'V73': 'float32', 'V74': 'float32', 'V75': 'float32', 'V76': 'float32', 'V77': 'float32', 'V78': 'float32', 'V79': 'float32', 'V80': 'float32', 'V81': 'float32', 'V82': 'float32', 'V83': 'float32', 'V84': 'float32', 'V85': 'float32', 'V86': 'float32', 'V87': 'float32', 'V88': 'float32', 'V89': 'float32', 'V90': 'float32', 'V91': 'float32', 'V92': 'float32', 'V93': 'float32', 'V94': 'float32', 'V95': 'float32', 'V96': 'float32', 'V97': 'float32', 'V98': 'float32', 'V99': 'float32', 'V100': 'float32', 'V101': 'float32', 'V102': 'float32', 'V103': 'float32', 'V104': 'float32', 'V105': 'float32', 'V106': 'float32', 'V107': 'float32', 'V108': 'float32', 'V109': 'float32', 'V110': 'float32', 'V111': 'float32', 'V112': 'float32', 'V113': 'float32', 'V114': 'float32', 'V115': 'float32', 'V116': 'float32', 'V117': 'float32', 'V118': 'float32', 'V119': 'float32', 'V120': 'float32', 'V121': 'float32', 'V122': 'float32', 'V123': 'float32', 'V124': 'float32', 'V125': 'float32', 'V126': 'float32', 'V127': 'float32', 'V128': 'float32', 'V129': 'float32', 'V130': 'float32', 'V131': 'float32', 'V132': 'float32', 'V133': 'float32', 'V134': 'float32', 'V135': 'float32', 'V136': 'float32', 'V137': 'float32', 'V138': 'float32', 'V139': 'float32', 'V140': 'float32', 'V141': 'float32', 'V142': 'float32', 'V143': 'float32', 'V144': 'float32', 'V145': 'float32', 'V146': 'float32', 'V147': 'float32', 'V148': 'float32', 'V149': 'float32', 'V150': 'float32', 'V151': 'float32', 'V152': 'float32', 'V153': 'float32', 'V154': 'float32', 'V155': 'float32', 'V156': 'float32', 'V157': 'float32', 'V158': 'float32', 'V159': 'float32', 'V160': 'float32', 'V161': 'float32', 'V162': 'float32', 'V163': 'float32', 'V164': 'float32', 'V165': 'float32', 'V166': 'float32', 'V167': 'float32', 'V168': 'float32', 'V169': 'float32', 'V170': 'float32', 'V171': 'float32', 'V172': 'float32', 'V173': 'float32', 'V174': 'float32', 'V175': 'float32', 'V176': 'float32', 'V177': 'float32', 'V178': 'float32', 'V179': 'float32', 'V180': 'float32', 'V181': 'float32', 'V182': 'float32', 'V183': 'float32', 'V184': 'float32', 'V185': 'float32', 'V186': 'float32', 'V187': 'float32', 'V188': 'float32', 'V189': 'float32', 'V190': 'float32', 'V191': 'float32', 'V192': 'float32', 'V193': 'float32', 'V194': 'float32', 'V195': 'float32', 'V196': 'float32', 'V197': 'float32', 'V198': 'float32', 'V199': 'float32', 'V200': 'float32', 'V201': 'float32', 'V202': 'float32', 'V203': 'float32', 'V204': 'float32', 'V205': 'float32', 'V206': 'float32', 'V207': 'float32', 'V208': 'float32', 'V209': 'float32', 'V210': 'float32', 'V211': 'float32', 'V212': 'float32', 'V213': 'float32', 'V214': 'float32', 'V215': 'float32', 'V216': 'float32', 'V217': 'float32', 'V218': 'float32', 'V219': 'float32', 'V220': 'float32', 'V221': 'float32', 'V222': 'float32', 'V223': 'float32', 'V224': 'float32', 'V225': 'float32', 'V226': 'float32', 'V227': 'float32', 'V228': 'float32', 'V229': 'float32', 'V230': 'float32', 'V231': 'float32', 'V232': 'float32', 'V233': 'float32', 'V234': 'float32', 'V235': 'float32', 'V236': 'float32', 'V237': 'float32', 'V238': 'float32', 'V239': 'float32', 'V240': 'float32', 'V241': 'float32', 'V242': 'float32', 'V243': 'float32', 'V244': 'float32', 'V245': 'float32', 'V246': 'float32', 'V247': 'float32', 'V248': 'float32', 'V249': 'float32', 'V250': 'float32', 'V251': 'float32', 'V252': 'float32', 'V253': 'float32', 'V254': 'float32', 'V255': 'float32', 'V256': 'float32', 'V257': 'float32', 'V258': 'float32', 'V259': 'float32', 'V260': 'float32', 'V261': 'float32', 'V262': 'float32', 'V263': 'float32', 'V264': 'float32', 'V265': 'float32', 'V266': 'float32', 'V267': 'float32', 'V268': 'float32', 'V269': 'float32', 'V270': 'float32', 'V271': 'float32', 'V272': 'float32', 'V273': 'float32', 'V274': 'float32', 'V275': 'float32', 'V276': 'float32', 'V277': 'float32', 'V278': 'float32', 'V279': 'float32', 'V280': 'float32', 'V281': 'float32', 'V282': 'float32', 'V283': 'float32', 'V284': 'float32', 'V285': 'float32', 'V286': 'float32', 'V287': 'float32', 'V288': 'float32', 'V289': 'float32', 'V290': 'float32', 'V291': 'float32', 'V292': 'float32', 'V293': 'float32', 'V294': 'float32', 'V295': 'float32', 'V296': 'float32', 'V297': 'float32', 'V298': 'float32', 'V299': 'float32', 'V300': 'float32', 'V301': 'float32', 'V302': 'float32', 'V303': 'float32', 'V304': 'float32', 'V305': 'float32', 'V306': 'float32', 'V307': 'float32', 'V308': 'float32', 'V309': 'float32', 'V310': 'float32', 'V311': 'float32', 'V312': 'float32', 'V313': 'float32', 'V314': 'float32', 'V315': 'float32', 'V316': 'float32', 'V317': 'float32', 'V318': 'float32', 'V319': 'float32', 'V320': 'float32', 'V321': 'float32', 'V322': 'float32', 'V323': 'float32', 'V324': 'float32', 'V325': 'float32', 'V326': 'float32', 'V327': 'float32', 'V328': 'float32', 'V329': 'float32', 'V330': 'float32', 'V331': 'float32', 'V332': 'float32', 'V333': 'float32', 'V334': 'float32', 'V335': 'float32', 'V336': 'float32', 'V337': 'float32', 'V338': 'float32', 'V339': 'float32', 'id_01': 'float32', 'id_02': 'float32', 'id_03': 'float32', 'id_04': 'float32', 'id_05': 'float32', 'id_06': 'float32', 'id_07': 'float32', 'id_08': 'float32', 'id_09': 'float32', 'id_10': 'float32', 'id_11': 'float32', 'id_12': 'category', 'id_13': 'float32', 'id_14': 'float32', 'id_15': 'category', 'id_16': 'category', 'id_17': 'float32', 'id_18': 'float32', 'id_19': 'float32', 'id_20': 'float32', 'id_21': 'float32', 'id_22': 'float32', 'id_23': 'category', 'id_24': 'float32', 'id_25': 'float32', 'id_26': 'float32', 'id_27': 'category', 'id_28': 'category', 'id_29': 'category', 'id_30': 'category', 'id_31': 'category', 'id_32': 'float32', 'id_33': 'category', 'id_34': 'category', 'id_35': 'category', 'id_36': 'category', 'id_37': 'category', 'id_38': 'category', 'DeviceType': 'category', 'DeviceInfo': 'category', 'TimeInDay': 'int32', 'Cents': 'float32'}\n", + "2024-05-29 13:32:05,981 pid:62817 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (32, 434) executed in 0:00:00.043595\n", + "Executed 'Accuracy on data slice “`D2` < 120.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}: \n", " Test failed\n", - " Metric: 0.84\n", + " Metric: 0.78\n", + " \n", " \n", - " \n" + "2024-05-29 13:32:05,999 pid:62817 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 13:32:05,999 pid:62817 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 13:32:05,999 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`D3` >= 13.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.6764705882352942}\n", + "2024-05-29 13:32:06,000 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`D5` >= 10.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.6774193548387096}\n", + "2024-05-29 13:32:06,000 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`D11` >= 31.000” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.7027027027027027}\n", + "2024-05-29 13:32:06,001 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`card1` < 12740.000 AND `card1` >= 7867.000” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.7096774193548387}\n", + "2024-05-29 13:32:06,003 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`M8` == \"F\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.71875}\n", + "2024-05-29 13:32:06,004 pid:62817 MainThread giskard.core.suite INFO Precision on data slice “`D15` >= 1.000 AND `D15` < 232.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}): {failed, metric=0.68}\n", + "2024-05-29 13:32:06,004 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`M7` == \"F\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.7380952380952381}\n", + "2024-05-29 13:32:06,005 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`D1` >= 0.500 AND `D1` < 120.000” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.7419354838709677}\n", + "2024-05-29 13:32:06,005 pid:62817 MainThread giskard.core.suite INFO Precision on data slice “`C9` >= 0.500 AND `C9` < 1.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}): {failed, metric=0.6923076923076923}\n", + "2024-05-29 13:32:06,005 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`V10` < 0.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.7714285714285715}\n", + "2024-05-29 13:32:06,006 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`V11` < 0.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.7714285714285715}\n", + "2024-05-29 13:32:06,008 pid:62817 MainThread giskard.core.suite INFO Precision on data slice “`P_emaildomain` == \"gmail.com\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}): {failed, metric=0.7142857142857143}\n", + "2024-05-29 13:32:06,008 pid:62817 MainThread giskard.core.suite INFO Precision on data slice “`M6` == \"T\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.7707547169811321}): {failed, metric=0.7142857142857143}\n", + "2024-05-29 13:32:06,008 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`C6` >= 1.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.775}\n", + "2024-05-29 13:32:06,009 pid:62817 MainThread giskard.core.suite INFO Accuracy on data slice “`D2` < 120.000” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.836}): {failed, metric=0.78125}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Precision on data slice “`D15` >= 4.000 AND `D15` < 344.500”\n
\n \n Measured Metric = 0.7\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `D15` >= 4.000 AND `D15` < 344.500\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`D4` >= 81.000”\n
\n \n Measured Metric = 0.75\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `D4` >= 81.000\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n
\n \n \n
\n Test Accuracy on data slice “`D2` >= 108.500”\n
\n \n Measured Metric = 0.79412\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `D2` >= 108.500\n
\n \n
\n threshold\n 0.8664000000000001\n
\n \n
\n
\n \n \n
\n Test Accuracy on data slice “`C6` >= 1.500”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `C6` >= 1.500\n
\n \n
\n threshold\n 0.8664000000000001\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`D11` >= 65.000”\n
\n \n Measured Metric = 0.80556\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `D11` >= 65.000\n
\n \n
\n threshold\n 0.8664000000000001\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`V310` >= 48.475”\n
\n \n Measured Metric = 0.78571\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `V310` >= 48.475\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`C11` >= 1.500”\n
\n \n Measured Metric = 0.79167\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `C11` >= 1.500\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`D5` >= 13.500”\n
\n \n Measured Metric = 0.8125\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `D5` >= 13.500\n
\n \n
\n threshold\n 0.8664000000000001\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`C1` >= 2.500”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `C1` >= 2.500\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`V283` < 0.500”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `V283` < 0.500\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`V282` < 0.500”\n
\n \n Measured Metric = 0.8\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `V282` < 0.500\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`D3` >= 13.500”\n
\n \n Measured Metric = 0.82353\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `D3` >= 13.500\n
\n \n
\n threshold\n 0.8664000000000001\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`V285` >= 0.500”\n
\n \n Measured Metric = 0.80645\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `V285` >= 0.500\n
\n \n
\n threshold\n 0.8444444444444444\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`addr1` >= 312.500”\n
\n \n Measured Metric = 0.83333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `addr1` >= 312.500\n
\n \n
\n threshold\n 0.8603773584905661\n
\n \n
\n \n \n \n
\n Test Accuracy on data slice “`D10` >= 222.000”\n
\n \n Measured Metric = 0.84375\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n af2b2aaa-c0d6-4428-9a9c-9aa288c718f4\n
\n \n
\n dataset\n fraud_detection_adversarial_dataset\n
\n \n
\n slicing_function\n `D10` >= 222.000\n
\n \n
\n threshold\n 0.8664000000000001\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`D3` >= 13.500”\n", + "
\n", + " \n", + " Measured Metric = 0.67647\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `D3` >= 13.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`D5` >= 10.500”\n", + "
\n", + " \n", + " Measured Metric = 0.67742\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `D5` >= 10.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`D11` >= 31.000”\n", + "
\n", + " \n", + " Measured Metric = 0.7027\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `D11` >= 31.000\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`card1` < 12740.000 AND `card1` >= 7867.000”\n", + "
\n", + " \n", + " Measured Metric = 0.70968\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `card1` < 12740.000 AND `card1` >= 7867.000\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`M8` == "F"”\n", + "
\n", + " \n", + " Measured Metric = 0.71875\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `M8` == "F"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`D15` >= 1.000 AND `D15` < 232.500”\n", + "
\n", + " \n", + " Measured Metric = 0.68\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `D15` >= 1.000 AND `D15` < 232.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7707547169811321\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`M7` == "F"”\n", + "
\n", + " \n", + " Measured Metric = 0.7381\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `M7` == "F"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`D1` >= 0.500 AND `D1` < 120.000”\n", + "
\n", + " \n", + " Measured Metric = 0.74194\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `D1` >= 0.500 AND `D1` < 120.000\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`C9` >= 0.500 AND `C9` < 1.500”\n", + "
\n", + " \n", + " Measured Metric = 0.69231\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `C9` >= 0.500 AND `C9` < 1.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7707547169811321\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`V10` < 0.500”\n", + "
\n", + " \n", + " Measured Metric = 0.77143\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `V10` < 0.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`V11` < 0.500”\n", + "
\n", + " \n", + " Measured Metric = 0.77143\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `V11` < 0.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`P_emaildomain` == "gmail.com"”\n", + "
\n", + " \n", + " Measured Metric = 0.71429\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `P_emaildomain` == "gmail.com"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7707547169811321\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`M6` == "T"”\n", + "
\n", + " \n", + " Measured Metric = 0.71429\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `M6` == "T"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.7707547169811321\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`C6` >= 1.500”\n", + "
\n", + " \n", + " Measured Metric = 0.775\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `C6` >= 1.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Accuracy on data slice “`D2` < 120.000”\n", + "
\n", + " \n", + " Measured Metric = 0.78125\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " train_test_data_classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " fraud_detection_adversarial_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `D2` < 120.000\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.836\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 13, "metadata": {}, @@ -675,118 +3972,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to: \n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -805,7 +3990,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.6" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/insurance_prediction_lgbm.ipynb b/docs/reference/notebooks/insurance_prediction_lgbm.ipynb index 6c03460669..162e15d080 100644 --- a/docs/reference/notebooks/insurance_prediction_lgbm.ipynb +++ b/docs/reference/notebooks/insurance_prediction_lgbm.ipynb @@ -22,12 +22,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -55,28 +50,28 @@ }, { "cell_type": "markdown", - "source": [ - "We also install the project-specific dependencies for this tutorial." - ], + "id": "b831a269a6537938", "metadata": { "collapsed": false }, - "id": "b831a269a6537938" + "source": [ + "We also install the project-specific dependencies for this tutorial." + ] }, { "cell_type": "code", "execution_count": null, - "outputs": [], - "source": [ - "%pip install lightgbm" - ], + "id": "797dd8f836bd3a03", "metadata": { - "collapsed": false, "ExecuteTime": { "start_time": "2023-08-01T15:21:01.831520Z" - } + }, + "collapsed": false }, - "id": "797dd8f836bd3a03" + "outputs": [], + "source": [ + "%pip install lightgbm" + ] }, { "cell_type": "markdown", @@ -105,11 +100,11 @@ "execution_count": 1, "id": "3c8f7165dcf10fbe", "metadata": { - "hidden": true, "ExecuteTime": { "end_time": "2023-11-09T12:44:00.915388Z", "start_time": "2023-11-09T12:43:55.199949Z" - } + }, + "hidden": true }, "outputs": [], "source": [ @@ -126,35 +121,35 @@ "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n", "\n", - "from giskard import GiskardClient, testing, Dataset, Model, scan, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { "cell_type": "markdown", - "source": [ - "## Notebook-level settings" - ], + "id": "86bca9ebc5a47498", "metadata": { "collapsed": false }, - "id": "86bca9ebc5a47498" + "source": [ + "## Notebook-level settings" + ] }, { "cell_type": "code", "execution_count": 2, - "outputs": [], - "source": [ - "logging.set_verbosity(logging.ERROR)\n", - "warnings.filterwarnings(\"ignore\", message=r\"Passing\", category=FutureWarning)" - ], + "id": "7024ed0a", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:44:01.669179Z", "start_time": "2023-11-09T12:44:01.634934Z" - } + }, + "collapsed": false }, - "id": "7024ed0a" + "outputs": [], + "source": [ + "logging.set_verbosity(logging.ERROR)\n", + "warnings.filterwarnings(\"ignore\", message=r\"Passing\", category=FutureWarning)" + ] }, { "cell_type": "markdown", @@ -171,11 +166,11 @@ "execution_count": 3, "id": "434314c0d4cf31fb", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:44:02.296748Z", "start_time": "2023-11-09T12:44:02.274216Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -213,11 +208,11 @@ "execution_count": 4, "id": "fe19935c186fd365", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:44:03.389591Z", "start_time": "2023-11-09T12:44:03.347340Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -265,11 +260,11 @@ "execution_count": 6, "id": "2a5351b1c97f2a31", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:44:04.230046Z", "start_time": "2023-11-09T12:44:04.204640Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -289,14 +284,14 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "id": "4d95d45185280742", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:44:04.870629Z", "start_time": "2023-11-09T12:44:04.828841Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -336,11 +331,11 @@ "execution_count": 8, "id": "64d42c05c8107e59", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T12:44:05.597336Z", "start_time": "2023-11-09T12:44:05.574669Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -400,7 +395,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Wrap the prediction function so that the whole pipeline is saved to the Hub\n", + "# Wrap the prediction function\n", "def prediction_function(df):\n", " return pipeline.predict(df)\n", "\n", @@ -422,13 +417,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], + "id": "8dc6d9def6fcfe66", "metadata": { "collapsed": false }, - "id": "8dc6d9def6fcfe66" + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -465,7 +460,1115 @@ "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -485,15 +1588,15 @@ }, { "cell_type": "markdown", + "id": "e94f621a1e7fb82d", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - }, - "id": "e94f621a1e7fb82d" + ] }, { "cell_type": "code", @@ -510,32 +1613,335 @@ "name": "stdout", "output_type": "stream", "text": [ - "Executed 'MSE on data slice “`region` == \"northeast\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", + "2024-05-29 13:33:44,636 pid:63742 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'} to {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'}\n", + "2024-05-29 13:33:44,638 pid:63742 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (96, 7) executed in 0:00:00.010454\n", + "Executed 'MSE on data slice “`region` == \"northeast\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", " Test failed\n", " Metric: 20989697.45\n", " \n", " \n", - "Executed 'MSE on data slice “`sex` == \"female\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", + "2024-05-29 13:33:44,653 pid:63742 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'} to {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'}\n", + "2024-05-29 13:33:44,655 pid:63742 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (153, 7) executed in 0:00:00.008035\n", + "Executed 'MSE on data slice “`sex` == \"female\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", " Test failed\n", " Metric: 18686445.91\n", " \n", " \n", - "Executed 'MSE on data slice “`smoker` == \"no\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", + "2024-05-29 13:33:44,666 pid:63742 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'} to {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'}\n", + "2024-05-29 13:33:44,669 pid:63742 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (264, 7) executed in 0:00:00.007542\n", + "Executed 'MSE on data slice “`smoker` == \"no\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", " Test failed\n", " Metric: 18440360.15\n", " \n", " \n", - "Executed 'MSE on data slice “`region` == \"southeast\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", + "2024-05-29 13:33:44,677 pid:63742 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'} to {'age': 'int64', 'sex': 'object', 'bmi': 'float64', 'children': 'int64', 'smoker': 'object', 'region': 'object'}\n", + "2024-05-29 13:33:44,678 pid:63742 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (88, 7) executed in 0:00:00.004697\n", + "Executed 'MSE on data slice “`region` == \"southeast\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}: \n", " Test failed\n", " Metric: 17201661.08\n", " \n", - " \n" + " \n", + "2024-05-29 13:33:44,682 pid:63742 MainThread giskard.core.suite INFO Executed test suite 'Test suite'\n", + "2024-05-29 13:33:44,682 pid:63742 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 13:33:44,683 pid:63742 MainThread giskard.core.suite INFO MSE on data slice “`region` == \"northeast\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}): {failed, metric=20989697.452960182}\n", + "2024-05-29 13:33:44,683 pid:63742 MainThread giskard.core.suite INFO MSE on data slice “`sex` == \"female\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}): {failed, metric=18686445.912572958}\n", + "2024-05-29 13:33:44,683 pid:63742 MainThread giskard.core.suite INFO MSE on data slice “`smoker` == \"no\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}): {failed, metric=18440360.147996694}\n", + "2024-05-29 13:33:44,684 pid:63742 MainThread giskard.core.suite INFO MSE on data slice “`region` == \"southeast\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 17058376.93295317}): {failed, metric=17201661.078229036}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test MSE on data slice “`region` == "northeast"”\n
\n \n Measured Metric = 20989697.45296\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n d2f80988-54e1-404c-abc0-09a1bfc81298\n
\n \n
\n dataset\n insurance dataset\n
\n \n
\n slicing_function\n `region` == "northeast"\n
\n \n
\n threshold\n 17058376.93295317\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`sex` == "female"”\n
\n \n Measured Metric = 18686445.91257\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n d2f80988-54e1-404c-abc0-09a1bfc81298\n
\n \n
\n dataset\n insurance dataset\n
\n \n
\n slicing_function\n `sex` == "female"\n
\n \n
\n threshold\n 17058376.93295317\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`smoker` == "no"”\n
\n \n Measured Metric = 18440360.148\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n d2f80988-54e1-404c-abc0-09a1bfc81298\n
\n \n
\n dataset\n insurance dataset\n
\n \n
\n slicing_function\n `smoker` == "no"\n
\n \n
\n threshold\n 17058376.93295317\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`region` == "southeast"”\n
\n \n Measured Metric = 17201661.07823\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n d2f80988-54e1-404c-abc0-09a1bfc81298\n
\n \n
\n dataset\n insurance dataset\n
\n \n
\n slicing_function\n `region` == "southeast"\n
\n \n
\n threshold\n 17058376.93295317\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`region` == "northeast"”\n", + "
\n", + " \n", + " Measured Metric = 20989697.45296\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " insurance model\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " insurance dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `region` == "northeast"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 17058376.93295317\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`sex` == "female"”\n", + "
\n", + " \n", + " Measured Metric = 18686445.91257\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " insurance model\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " insurance dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `sex` == "female"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 17058376.93295317\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`smoker` == "no"”\n", + "
\n", + " \n", + " Measured Metric = 18440360.148\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " insurance model\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " insurance dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `smoker` == "no"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 17058376.93295317\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`region` == "southeast"”\n", + "
\n", + " \n", + " Measured Metric = 17201661.07823\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " insurance model\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " insurance dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `region` == "southeast"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 17058376.93295317\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 13, "metadata": {}, @@ -578,124 +1984,6 @@ "source": [ "test_suite.add_test(testing.test_rmse(model=giskard_model, dataset=giskard_dataset, threshold=10.0)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - }, - "id": "20b6dd6e50c9f9b6" - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - }, - "id": "a6259312ce27b32" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - }, - "id": "8430cb1806cb0b36" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - }, - "id": "7984f79517cb379f" - }, - { - "cell_type": "markdown", - "id": "4a31adf6", - "metadata": {}, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3e139194b2d1076", - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - " \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ], - "metadata": { - "collapsed": false - }, - "id": "dedf0195ce8a1a6a" - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - }, - "id": "4e2b05cbe891bce0" } ], "metadata": { @@ -714,7 +2002,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.12" + "version": "3.10.14" }, "toc": { "base_numbering": "0", diff --git a/docs/reference/notebooks/m5_sales_prediction_lgbm.ipynb b/docs/reference/notebooks/m5_sales_prediction_lgbm.ipynb index d17986afbd..06e8e45a86 100644 --- a/docs/reference/notebooks/m5_sales_prediction_lgbm.ipynb +++ b/docs/reference/notebooks/m5_sales_prediction_lgbm.ipynb @@ -19,12 +19,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -39,7 +34,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2023-08-24T10:39:48.395480Z", @@ -63,14 +58,14 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 1, "metadata": { - "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", - "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5", "ExecuteTime": { "end_time": "2023-11-09T13:52:15.838102Z", "start_time": "2023-11-09T13:52:07.960231Z" - } + }, + "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", + "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5" }, "outputs": [], "source": [ @@ -83,7 +78,7 @@ "from lightgbm import LGBMRegressor\n", "from sklearn.metrics import r2_score\n", "\n", - "from giskard import Dataset, Model, testing, GiskardClient, scan, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { @@ -97,13 +92,13 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T13:52:18.124518Z", "start_time": "2023-11-09T13:52:18.073663Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -276,13 +271,13 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T13:53:18.825367Z", "start_time": "2023-11-09T13:53:18.772930Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -369,12 +364,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -400,18 +395,1123 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 10, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T13:54:20.338407Z", "start_time": "2023-11-09T13:54:20.146169Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -432,58 +1532,361 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T13:54:32.233432Z", "start_time": "2023-11-09T13:54:32.004498Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'MSE on data slice “`snap_TX` == 1”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", + "2024-05-29 13:43:58,662 pid:66104 MainThread giskard.datasets.base INFO Casting dataframe columns from {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'} to {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'}\n", + "2024-05-29 13:43:58,664 pid:66104 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1000, 15) executed in 0:00:00.015511\n", + "Executed 'MSE on data slice “`snap_TX` == 1”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", " Test failed\n", " Metric: 8.92\n", " \n", " \n", - "Executed 'MSE on data slice “`snap_WI` == 1”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", + "2024-05-29 13:43:58,683 pid:66104 MainThread giskard.datasets.base INFO Casting dataframe columns from {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'} to {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'}\n", + "2024-05-29 13:43:58,685 pid:66104 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1000, 15) executed in 0:00:00.012052\n", + "Executed 'MSE on data slice “`snap_WI` == 1”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", " Test failed\n", " Metric: 8.64\n", " \n", " \n", - "Executed 'MSE on data slice “`wm_yr_wk` == 1.161e+04”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", + "2024-05-29 13:43:58,699 pid:66104 MainThread giskard.datasets.base INFO Casting dataframe columns from {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'} to {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'}\n", + "2024-05-29 13:43:58,701 pid:66104 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (700, 15) executed in 0:00:00.009145\n", + "Executed 'MSE on data slice “`wm_yr_wk` == 1.161e+04”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", " Test failed\n", " Metric: 8.63\n", " \n", " \n", - "Executed 'MSE on data slice “`snap_CA` == 1”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", + "2024-05-29 13:43:58,716 pid:66104 MainThread giskard.datasets.base INFO Casting dataframe columns from {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'} to {'item_id': 'int64', 'dept_id': 'int64', 'cat_id': 'int64', 'store_id': 'int64', 'state_id': 'int64', 'wm_yr_wk': 'int64', 'event_name_1': 'int64', 'event_type_1': 'int64', 'event_name_2': 'int64', 'event_type_2': 'int64', 'snap_CA': 'int64', 'snap_TX': 'int64', 'snap_WI': 'int64', 'sell_price': 'float64'}\n", + "2024-05-29 13:43:58,718 pid:66104 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1000, 15) executed in 0:00:00.009801\n", + "Executed 'MSE on data slice “`snap_CA` == 1”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}: \n", " Test failed\n", " Metric: 7.92\n", " \n", - " \n" + " \n", + "2024-05-29 13:43:58,723 pid:66104 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 13:43:58,723 pid:66104 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 13:43:58,724 pid:66104 MainThread giskard.core.suite INFO MSE on data slice “`snap_TX` == 1” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}): {failed, metric=8.915211806609342}\n", + "2024-05-29 13:43:58,724 pid:66104 MainThread giskard.core.suite INFO MSE on data slice “`snap_WI` == 1” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}): {failed, metric=8.637798447295712}\n", + "2024-05-29 13:43:58,725 pid:66104 MainThread giskard.core.suite INFO MSE on data slice “`wm_yr_wk` == 1.161e+04” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}): {failed, metric=8.630063641271192}\n", + "2024-05-29 13:43:58,725 pid:66104 MainThread giskard.core.suite INFO MSE on data slice “`snap_CA` == 1” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 7.1383301822423295}): {failed, metric=7.922860008805299}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test MSE on data slice “`snap_TX` == 1”\n
\n \n Measured Metric = 8.91521\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 953d4ee7-82dc-421e-910d-9043450544ae\n
\n \n
\n dataset\n M5 products timeseries dataset\n
\n \n
\n slicing_function\n `snap_TX` == 1\n
\n \n
\n threshold\n 7.1383301822423295\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`snap_WI` == 1”\n
\n \n Measured Metric = 8.6378\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 953d4ee7-82dc-421e-910d-9043450544ae\n
\n \n
\n dataset\n M5 products timeseries dataset\n
\n \n
\n slicing_function\n `snap_WI` == 1\n
\n \n
\n threshold\n 7.1383301822423295\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`wm_yr_wk` == 1.161e+04”\n
\n \n Measured Metric = 8.63006\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 953d4ee7-82dc-421e-910d-9043450544ae\n
\n \n
\n dataset\n M5 products timeseries dataset\n
\n \n
\n slicing_function\n `wm_yr_wk` == 1.161e+04\n
\n \n
\n threshold\n 7.1383301822423295\n
\n \n
\n
\n \n \n
\n Test MSE on data slice “`snap_CA` == 1”\n
\n \n Measured Metric = 7.92286\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 953d4ee7-82dc-421e-910d-9043450544ae\n
\n \n
\n dataset\n M5 products timeseries dataset\n
\n \n
\n slicing_function\n `snap_CA` == 1\n
\n \n
\n threshold\n 7.1383301822423295\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`snap_TX` == 1”\n", + "
\n", + " \n", + " Measured Metric = 8.91521\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " M5 sales timeseries regressor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " M5 products timeseries dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `snap_TX` == 1\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 7.1383301822423295\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`snap_WI` == 1”\n", + "
\n", + " \n", + " Measured Metric = 8.6378\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " M5 sales timeseries regressor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " M5 products timeseries dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `snap_WI` == 1\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 7.1383301822423295\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`wm_yr_wk` == 1.161e+04”\n", + "
\n", + " \n", + " Measured Metric = 8.63006\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " M5 sales timeseries regressor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " M5 products timeseries dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `wm_yr_wk` == 1.161e+04\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 7.1383301822423295\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test MSE on data slice “`snap_CA` == 1”\n", + "
\n", + " \n", + " Measured Metric = 7.92286\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " M5 sales timeseries regressor\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " M5 products timeseries dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `snap_CA` == 1\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 7.1383301822423295\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 12, + "execution_count": 11, "metadata": {}, "output_type": "execute_result" } @@ -522,118 +1925,6 @@ "source": [ "test_suite.add_test(testing.test_r2(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to: \n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ] - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - "\n", - "* Check for regressions after training a new model\n", - "* Automate the test suite execution in a CI/CD pipeline\n", - "* Compare several models during the prototyping phase" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -652,7 +1943,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.6" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/medical_transcript_classification_sklearn.ipynb b/docs/reference/notebooks/medical_transcript_classification_sklearn.ipynb index b18c318240..4694220c18 100644 --- a/docs/reference/notebooks/medical_transcript_classification_sklearn.ipynb +++ b/docs/reference/notebooks/medical_transcript_classification_sklearn.ipynb @@ -22,12 +22,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -55,13 +50,13 @@ }, { "cell_type": "markdown", - "source": [ - "We also install the project-specific dependencies for this tutorial." - ], + "id": "a722bc090c468228", "metadata": { "collapsed": false }, - "id": "a722bc090c468228" + "source": [ + "We also install the project-specific dependencies for this tutorial." + ] }, { "cell_type": "code", @@ -87,14 +82,14 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 1, "id": "14e64fb17dd952c", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:19:23.444151Z", "start_time": "2023-11-09T14:19:23.414422Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -114,7 +109,7 @@ "from sklearn.preprocessing import FunctionTransformer\n", "from typing import Iterable\n", "\n", - "from giskard import Dataset, Model, scan, GiskardClient, testing, Suite" + "from giskard import Dataset, Model, scan, testing" ] }, { @@ -129,14 +124,14 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 2, "id": "eb6ddf97e5bfaa17", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:19:24.265761Z", "start_time": "2023-11-09T14:19:24.208447Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -175,26 +170,26 @@ }, { "cell_type": "markdown", - "source": [ - "### Download NLTK stopwords corpus" - ], + "id": "51ae2ea76ba01e93", "metadata": { "collapsed": false }, - "id": "51ae2ea76ba01e93" + "source": [ + "### Download NLTK stopwords corpus" + ] }, { "cell_type": "code", "execution_count": null, + "id": "54d3dabd4484064e", + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "# Download list of english stopwords.\n", "nltk.download('stopwords')" - ], - "metadata": { - "collapsed": false - }, - "id": "54d3dabd4484064e" + ] }, { "cell_type": "markdown", @@ -208,14 +203,14 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 4, "id": "2016f55da2fb2636", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:19:26.322440Z", "start_time": "2023-11-09T14:19:26.198889Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -276,14 +271,14 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 6, "id": "ef066b868f02dea0", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:19:27.644994Z", "start_time": "2023-11-09T14:19:27.618840Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -305,14 +300,14 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": null, "id": "2eadc4944d498729", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:19:28.691833Z", "start_time": "2023-11-09T14:19:28.637886Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -347,14 +342,14 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 8, "id": "cc4c51a3519004b1", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:19:29.581720Z", "start_time": "2023-11-09T14:19:29.501498Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -449,7 +444,7 @@ }, "outputs": [], "source": [ - "# Wrap the prediction function so that the whole pipeline get saved to the Hub \n", + "# Wrap the prediction function\n", "def prediction_function(df):\n", " return pipeline.predict_proba(df)\n", "\n", @@ -469,13 +464,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], + "id": "c6eb3203bcda614e", "metadata": { "collapsed": false }, - "id": "c6eb3203bcda614e" + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", @@ -503,19 +498,4430 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 12, "id": "eb4a2acdff290603", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:35:48.509583Z", "start_time": "2023-11-09T14:35:48.257791Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -527,13 +4933,13 @@ }, { "cell_type": "markdown", - "source": [ - "## Generate comprehensive test suites automatically for your model" - ], + "id": "72547cc27176d759", "metadata": { "collapsed": false }, - "id": "72547cc27176d759" + "source": [ + "## Generate comprehensive test suites automatically for your model" + ] }, { "cell_type": "markdown", @@ -549,183 +4955,1812 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 14, "id": "e740cee558970a9c", "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:34:00.381087Z", "start_time": "2023-11-09T14:33:54.189523Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 13:49:53,802 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:53,804 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (371, 2) executed in 0:00:00.009591\n", + "2024-05-29 13:49:54,057 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,048 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (371, 2) executed in 0:00:00.998800\n", + "2024-05-29 13:49:55,051 pid:66538 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:01.259399\n", + "2024-05-29 13:49:55,051 pid:66538 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.000281\n", + "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", " Test failed\n", - " Metric: 0.91\n", - " - [TestMessageLevel.INFO] 371 rows were perturbed\n", - " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"weight\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", - " Test failed\n", - " Metric: 0.74\n", - " \n", + " Metric: 0.9\n", + " - [INFO] 371 rows were perturbed\n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"having\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,070 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,071 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (60, 2) executed in 0:00:00.005307\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"temperature\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.74\n", + " Metric: 0.73\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"today\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,090 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,091 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (97, 2) executed in 0:00:00.006231\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"dr\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.74\n", + " Metric: 0.73\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"temperature\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,111 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,112 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (76, 2) executed in 0:00:00.007671\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"weight\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.73\n", + " Metric: 0.72\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"dr\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,132 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,132 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (65, 2) executed in 0:00:00.006201\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"having\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.73\n", + " Metric: 0.72\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"follow\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,151 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,152 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (71, 2) executed in 0:00:00.005367\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"today\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", " Metric: 0.72\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`avg_whitespace(transcription)` >= 0.160”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,172 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,173 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (80, 2) executed in 0:00:00.005348\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"follow\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.71\n", + " Metric: 0.7\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"blood\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,191 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,192 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (191, 2) executed in 0:00:00.008549\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"blood\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", " Metric: 0.7\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"distress\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,213 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,214 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (70, 2) executed in 0:00:00.007286\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"distress\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", " Metric: 0.69\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"mg\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,236 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,237 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (126, 2) executed in 0:00:00.007065\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"stable\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.69\n", + " Metric: 0.68\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"continue\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,257 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,258 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (93, 2) executed in 0:00:00.006464\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"mg\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.69\n", + " Metric: 0.68\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"stable\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,270 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,271 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (214, 2) executed in 0:00:00.009823\n", + "Executed 'Overconfidence on data slice “`text_length(transcription)` >= 2145.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", " Metric: 0.68\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`avg_word_length(transcription)` < 5.789”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,291 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,292 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (85, 2) executed in 0:00:00.006814\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"discharge\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.68\n", + " Metric: 0.67\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`text_length(transcription)` >= 2145.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,312 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,313 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (101, 2) executed in 0:00:00.007278\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"hospital\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", - " Metric: 0.68\n", + " Metric: 0.67\n", + " \n", + " \n", + "2024-05-29 13:49:55,334 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,335 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (52, 2) executed in 0:00:00.006081\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"continue\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", + " Test failed\n", + " Metric: 0.67\n", " \n", " \n", - "Executed 'Overconfidence on data slice “`transcription` contains \"discharge\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6569444444444444, 'p_threshold': 0.2468526289804987}: \n", + "2024-05-29 13:49:55,354 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,355 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (76, 2) executed in 0:00:00.005812\n", + "Executed 'Overconfidence on data slice “`transcription` contains \"vital\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}: \n", " Test failed\n", " Metric: 0.67\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"xyz\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,374 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,375 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (22, 2) executed in 0:00:00.005195\n", + "Executed 'Precision on data slice “`transcription` contains \"xyz\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.32\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"subjective\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,397 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,398 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (30, 2) executed in 0:00:00.004141\n", + "Executed 'Precision on data slice “`transcription` contains \"subjective\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.37\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"admission\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,417 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,418 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (64, 2) executed in 0:00:00.005701\n", + "Executed 'Precision on data slice “`transcription` contains \"admission\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.38\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"daily\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,438 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,439 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (65, 2) executed in 0:00:00.005455\n", + "Executed 'Precision on data slice “`transcription` contains \"daily\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.38\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"coronary\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,460 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,462 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.006355\n", + "Executed 'Precision on data slice “`transcription` contains \"coronary\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", + " Test failed\n", + " Metric: 0.39\n", + " \n", + " \n", + "2024-05-29 13:49:55,482 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,484 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.005071\n", + "Executed 'Precision on data slice “`transcription` contains \"aspirin\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.39\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"followup\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,505 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,506 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (48, 2) executed in 0:00:00.005088\n", + "Executed 'Precision on data slice “`transcription` contains \"followup\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"lung\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,526 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,528 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (20, 2) executed in 0:00:00.005684\n", + "Executed 'Precision on data slice “`transcription` contains \"lung\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"count\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,549 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,550 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (50, 2) executed in 0:00:00.006254\n", + "Executed 'Precision on data slice “`transcription` contains \"count\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"function\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,581 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,593 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (29, 2) executed in 0:00:00.016641\n", + "Executed 'Precision on data slice “`transcription` contains \"function\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.41\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"abc\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,648 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,649 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (24, 2) executed in 0:00:00.008552\n", + "Executed 'Precision on data slice “`transcription` contains \"abc\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.42\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"improved\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,679 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,681 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (31, 2) executed in 0:00:00.007550\n", + "Executed 'Precision on data slice “`transcription` contains \"improved\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.42\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"discharge\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,703 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,704 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (52, 2) executed in 0:00:00.006033\n", + "Executed 'Precision on data slice “`transcription` contains \"continue\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.42\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"greater\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,725 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,726 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (85, 2) executed in 0:00:00.006033\n", + "Executed 'Precision on data slice “`transcription` contains \"discharge\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", - " Metric: 0.43\n", + " Metric: 0.42\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"aspirin\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", + "2024-05-29 13:49:55,862 pid:66538 MainThread giskard.datasets.base INFO Casting dataframe columns from {'transcription': 'object'} to {'transcription': 'object'}\n", + "2024-05-29 13:49:55,863 pid:66538 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (23, 2) executed in 0:00:00.005574\n", + "Executed 'Precision on data slice “`transcription` contains \"greater\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}: \n", " Test failed\n", " Metric: 0.43\n", " \n", " \n", - "Executed 'Precision on data slice “`transcription` contains \"continue\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.581266846361186}: \n", - " Test failed\n", - " Metric: 0.44\n", - " \n", - " \n" + "2024-05-29 13:49:55,866 pid:66538 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 13:49:55,866 pid:66538 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 13:49:55,866 pid:66538 MainThread giskard.core.suite INFO Invariance to “Add typos” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.9029649595687331}\n", + "2024-05-29 13:49:55,867 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"temperature\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.7333333333333333}\n", + "2024-05-29 13:49:55,867 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"dr\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.7254901960784313}\n", + "2024-05-29 13:49:55,867 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"weight\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.71875}\n", + "2024-05-29 13:49:55,867 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"having\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.71875}\n", + "2024-05-29 13:49:55,868 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"today\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.717948717948718}\n", + "2024-05-29 13:49:55,868 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"follow\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.7027027027027027}\n", + "2024-05-29 13:49:55,869 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"blood\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.6973684210526315}\n", + "2024-05-29 13:49:55,869 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"distress\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.6944444444444444}\n", + "2024-05-29 13:49:55,870 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"stable\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.6808510638297872}\n", + "2024-05-29 13:49:55,870 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"mg\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.68}\n", + "2024-05-29 13:49:55,870 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`text_length(transcription)` >= 2145.000” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.6756756756756757}\n", + "2024-05-29 13:49:55,870 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"discharge\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.673469387755102}\n", + "2024-05-29 13:49:55,871 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"hospital\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.6666666666666666}\n", + "2024-05-29 13:49:55,871 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"continue\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.6666666666666666}\n", + "2024-05-29 13:49:55,871 pid:66538 MainThread giskard.core.suite INFO Overconfidence on data slice “`transcription` contains \"vital\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.6524137931034483, 'p_threshold': 0.2468526289804987}): {failed, metric=0.6666666666666666}\n", + "2024-05-29 13:49:55,871 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"xyz\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.3181818181818182}\n", + "2024-05-29 13:49:55,872 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"subjective\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.36666666666666664}\n", + "2024-05-29 13:49:55,873 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"admission\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.375}\n", + "2024-05-29 13:49:55,873 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"daily\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.38461538461538464}\n", + "2024-05-29 13:49:55,873 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"coronary\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.391304347826087}\n", + "2024-05-29 13:49:55,873 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"aspirin\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.391304347826087}\n", + "2024-05-29 13:49:55,873 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"followup\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.3958333333333333}\n", + "2024-05-29 13:49:55,874 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"lung\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.4}\n", + "2024-05-29 13:49:55,874 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"count\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.4}\n", + "2024-05-29 13:49:55,875 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"function\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.41379310344827586}\n", + "2024-05-29 13:49:55,875 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"abc\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.4166666666666667}\n", + "2024-05-29 13:49:55,876 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"improved\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.41935483870967744}\n", + "2024-05-29 13:49:55,876 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"continue\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.4230769230769231}\n", + "2024-05-29 13:49:55,877 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"discharge\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.4235294117647059}\n", + "2024-05-29 13:49:55,878 pid:66538 MainThread giskard.core.suite INFO Precision on data slice “`transcription` contains \"greater\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5787061994609164}): {failed, metric=0.43478260869565216}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Invariance to “Add typos”\n
\n \n Measured Metric = 0.91375\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n transformation_function\n Add typos\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`transcription` contains "weight"”\n
\n \n Measured Metric = 0.74194\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "weight"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`transcription` contains "having"”\n
\n \n Measured Metric = 0.74194\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "having"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`transcription` contains "today"”\n
\n \n Measured Metric = 0.73684\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "today"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "temperature"”\n
\n \n Measured Metric = 0.73333\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "temperature"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "dr"”\n
\n \n Measured Metric = 0.72549\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "dr"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "follow"”\n
\n \n Measured Metric = 0.72222\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "follow"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`avg_whitespace(transcription)` >= 0.160”\n
\n \n Measured Metric = 0.71429\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `avg_whitespace(transcription)` >= 0.160\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "blood"”\n
\n \n Measured Metric = 0.69737\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "blood"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "distress"”\n
\n \n Measured Metric = 0.69444\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "distress"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "mg"”\n
\n \n Measured Metric = 0.69388\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "mg"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "continue"”\n
\n \n Measured Metric = 0.68966\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "continue"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "stable"”\n
\n \n Measured Metric = 0.68085\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "stable"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`avg_word_length(transcription)` < 5.789”\n
\n \n Measured Metric = 0.67647\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `avg_word_length(transcription)` < 5.789\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`text_length(transcription)` >= 2145.000”\n
\n \n Measured Metric = 0.67568\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `text_length(transcription)` >= 2145.000\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Overconfidence on data slice “`transcription` contains "discharge"”\n
\n \n Measured Metric = 0.67347\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "discharge"\n
\n \n
\n threshold\n 0.6569444444444444\n
\n \n
\n p_threshold\n 0.2468526289804987\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "xyz"”\n
\n \n Measured Metric = 0.31818\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "xyz"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "subjective"”\n
\n \n Measured Metric = 0.36667\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "subjective"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "admission"”\n
\n \n Measured Metric = 0.375\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "admission"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "daily"”\n
\n \n Measured Metric = 0.38462\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "daily"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "coronary"”\n
\n \n Measured Metric = 0.3913\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "coronary"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "followup"”\n
\n \n Measured Metric = 0.39583\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "followup"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "lung"”\n
\n \n Measured Metric = 0.4\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "lung"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "count"”\n
\n \n Measured Metric = 0.4\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "count"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "function"”\n
\n \n Measured Metric = 0.41379\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "function"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "abc"”\n
\n \n Measured Metric = 0.41667\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "abc"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "improved"”\n
\n \n Measured Metric = 0.41935\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "improved"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "discharge"”\n
\n \n Measured Metric = 0.42353\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "discharge"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "greater"”\n
\n \n Measured Metric = 0.43478\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "greater"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "aspirin"”\n
\n \n Measured Metric = 0.43478\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "aspirin"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`transcription` contains "continue"”\n
\n \n Measured Metric = 0.44231\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 082b3091-113e-4b56-b010-9902f9b9dcf8\n
\n \n
\n dataset\n medical_transcript_dataset\n
\n \n
\n slicing_function\n `transcription` contains "continue"\n
\n \n
\n threshold\n 0.581266846361186\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Add typos”\n", + "
\n", + " \n", + " Measured Metric = 0.90296\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Add typos\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "temperature"”\n", + "
\n", + " \n", + " Measured Metric = 0.73333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "temperature"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "dr"”\n", + "
\n", + " \n", + " Measured Metric = 0.72549\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "dr"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "weight"”\n", + "
\n", + " \n", + " Measured Metric = 0.71875\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "weight"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "having"”\n", + "
\n", + " \n", + " Measured Metric = 0.71875\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "having"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "today"”\n", + "
\n", + " \n", + " Measured Metric = 0.71795\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "today"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "follow"”\n", + "
\n", + " \n", + " Measured Metric = 0.7027\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "follow"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "blood"”\n", + "
\n", + " \n", + " Measured Metric = 0.69737\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "blood"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "distress"”\n", + "
\n", + " \n", + " Measured Metric = 0.69444\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "distress"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "stable"”\n", + "
\n", + " \n", + " Measured Metric = 0.68085\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "stable"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "mg"”\n", + "
\n", + " \n", + " Measured Metric = 0.68\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "mg"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`text_length(transcription)` >= 2145.000”\n", + "
\n", + " \n", + " Measured Metric = 0.67568\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text_length(transcription)` >= 2145.000\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "discharge"”\n", + "
\n", + " \n", + " Measured Metric = 0.67347\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "discharge"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "hospital"”\n", + "
\n", + " \n", + " Measured Metric = 0.66667\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "hospital"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "continue"”\n", + "
\n", + " \n", + " Measured Metric = 0.66667\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "continue"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`transcription` contains "vital"”\n", + "
\n", + " \n", + " Measured Metric = 0.66667\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "vital"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.6524137931034483\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.2468526289804987\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "xyz"”\n", + "
\n", + " \n", + " Measured Metric = 0.31818\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "xyz"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "subjective"”\n", + "
\n", + " \n", + " Measured Metric = 0.36667\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "subjective"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "admission"”\n", + "
\n", + " \n", + " Measured Metric = 0.375\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "admission"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "daily"”\n", + "
\n", + " \n", + " Measured Metric = 0.38462\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "daily"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "coronary"”\n", + "
\n", + " \n", + " Measured Metric = 0.3913\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "coronary"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "aspirin"”\n", + "
\n", + " \n", + " Measured Metric = 0.3913\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "aspirin"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "followup"”\n", + "
\n", + " \n", + " Measured Metric = 0.39583\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "followup"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "lung"”\n", + "
\n", + " \n", + " Measured Metric = 0.4\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "lung"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "count"”\n", + "
\n", + " \n", + " Measured Metric = 0.4\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "count"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "function"”\n", + "
\n", + " \n", + " Measured Metric = 0.41379\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "function"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "abc"”\n", + "
\n", + " \n", + " Measured Metric = 0.41667\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "abc"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "improved"”\n", + "
\n", + " \n", + " Measured Metric = 0.41935\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "improved"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "continue"”\n", + "
\n", + " \n", + " Measured Metric = 0.42308\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "continue"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "discharge"”\n", + "
\n", + " \n", + " Measured Metric = 0.42353\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "discharge"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`transcription` contains "greater"”\n", + "
\n", + " \n", + " Measured Metric = 0.43478\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " medical_transcript_classification\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " medical_transcript_dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `transcription` contains "greater"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5787061994609164\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 22, + "execution_count": 14, "metadata": {}, "output_type": "execute_result" } @@ -766,126 +6801,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "id": "79d44e8a70afe38b", - "metadata": { - "collapsed": false - }, - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ] - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - }, - "id": "750c995e5694fea7" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - }, - "id": "92a34e0f12453610" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - }, - "id": "f0e45c5b7f080e4c" - }, - { - "cell_type": "markdown", - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ], - "metadata": { - "collapsed": false - }, - "id": "8d18311f9990f724" - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2da07a815b6bcb09", - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ], - "metadata": { - "collapsed": false - }, - "id": "d7f52a115ae8d196" - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - }, - "id": "9a4aa5439a722398" } ], "metadata": { @@ -904,7 +6819,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.12" + "version": "3.10.14" }, "papermill": { "default_parameters": {}, diff --git a/docs/reference/notebooks/movie_review_sentiment_classification_pytorch_sklearn.ipynb b/docs/reference/notebooks/movie_review_sentiment_classification_pytorch_sklearn.ipynb index a5ccf6de34..3fe599fbcd 100644 --- a/docs/reference/notebooks/movie_review_sentiment_classification_pytorch_sklearn.ipynb +++ b/docs/reference/notebooks/movie_review_sentiment_classification_pytorch_sklearn.ipynb @@ -19,34 +19,29 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "## Install dependencies\n", "Make sure to install the `giskard`" - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", "execution_count": null, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "%pip install giskard --upgrade" - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "markdown", @@ -77,8 +72,7 @@ "from sklearn.metrics import accuracy_score\n", "from sklearn.model_selection import train_test_split\n", "\n", - "from giskard import Model, Dataset, scan, testing, Suite\n", - "from giskard.client.giskard_client import GiskardClient" + "from giskard import Model, Dataset, scan, testing" ] }, { @@ -94,11 +88,11 @@ "cell_type": "code", "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:37:37.115118Z", "start_time": "2023-11-09T14:37:37.074171Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -176,13 +170,13 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:37:40.551228Z", "start_time": "2023-11-09T14:37:40.473965Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -201,13 +195,13 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:37:41.496296Z", "start_time": "2023-11-09T14:37:41.374398Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -353,23 +347,23 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Scan your model for vulnerabilities with Giskard\n", "\n", "Giskard's scan allows you to detect vulnerabilities in your model automatically. These include performance biases, unrobustness, data leakage, stochasticity, underconfidence, ethical issues, and more. For detailed information about the scan feature, please refer to our [scan documentation](https://docs.giskard.ai/en/stable/open_source/scan/scan_nlp/index.html)." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", @@ -384,17 +378,768 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 11, "metadata": { - "collapsed": false, "ExecuteTime": { "start_time": "2023-11-09T14:43:30.984093Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -415,47 +1160,207 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 12, "metadata": { - "collapsed": false, "ExecuteTime": { "start_time": "2023-11-09T14:43:31.178727Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Overconfidence on data slice “`avg_whitespace(text)` >= 0.172”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4033333333333333, 'p_threshold': 0.5}: \n", - " Test failed\n", - " Metric: 0.43\n", - " \n", - " \n", - "Executed 'Precision on data slice “`text` contains \"movie\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.808091286307054}: \n", + "2024-05-29 13:52:50,519 pid:67935 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 13:52:50,520 pid:67935 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (62, 2) executed in 0:00:00.007627\n", + "Executed 'Precision on data slice “`text` contains \"movie\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8047520661157024}: \n", " Test failed\n", " Metric: 0.64\n", " \n", - " \n" + " \n", + "2024-05-29 13:52:50,530 pid:67935 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 13:52:50,536 pid:67935 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 13:52:50,543 pid:67935 MainThread giskard.core.suite INFO Precision on data slice “`text` contains \"movie\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.8047520661157024}): {failed, metric=0.6363636363636364}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`avg_whitespace(text)` >= 0.172”\n
\n \n Measured Metric = 0.43103\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 47a0c059-70ff-456b-802d-ac8faf08081f\n
\n \n
\n dataset\n Movie reviews dataset\n
\n \n
\n slicing_function\n `avg_whitespace(text)` >= 0.172\n
\n \n
\n threshold\n 0.4033333333333333\n
\n \n
\n p_threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Precision on data slice “`text` contains "movie"”\n
\n \n Measured Metric = 0.63636\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 47a0c059-70ff-456b-802d-ac8faf08081f\n
\n \n
\n dataset\n Movie reviews dataset\n
\n \n
\n slicing_function\n `text` contains "movie"\n
\n \n
\n threshold\n 0.808091286307054\n
\n \n
\n
\n \n
\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`text` contains "movie"”\n", + "
\n", + " \n", + " Measured Metric = 0.63636\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Movie reviews sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Movie reviews dataset\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `text` contains "movie"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.8047520661157024\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + "
\n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 11, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -494,118 +1399,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -624,7 +1417,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.6" + "version": "3.10.14" }, "widgets": { "application/vnd.jupyter.widget-state+json": { diff --git a/docs/reference/notebooks/newspaper_classification_pytorch.ipynb b/docs/reference/notebooks/newspaper_classification_pytorch.ipynb index f6c28962b9..76c5a720fa 100644 --- a/docs/reference/notebooks/newspaper_classification_pytorch.ipynb +++ b/docs/reference/notebooks/newspaper_classification_pytorch.ipynb @@ -21,12 +21,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -41,7 +36,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2023-08-22T12:41:11.380265Z", @@ -67,11 +62,11 @@ "cell_type": "code", "execution_count": 1, "metadata": { - "id": "eup4gpgVoA10", "ExecuteTime": { "end_time": "2023-11-09T14:48:28.531176Z", "start_time": "2023-11-09T14:48:19.132247Z" - } + }, + "id": "eup4gpgVoA10" }, "outputs": [], "source": [ @@ -89,7 +84,7 @@ "from torchtext.vocab import build_vocab_from_iterator\n", "from torchtext.data.functional import to_map_style_dataset\n", "\n", - "from giskard import Model, Dataset, GiskardClient, scan, testing, Suite" + "from giskard import Model, Dataset, scan, testing" ] }, { @@ -105,11 +100,11 @@ "cell_type": "code", "execution_count": 2, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:48:29.990594Z", "start_time": "2023-11-09T14:48:29.949237Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -144,11 +139,11 @@ "cell_type": "code", "execution_count": 3, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:48:31.034568Z", "start_time": "2023-11-09T14:48:31.006365Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -167,13 +162,13 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": null, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:48:31.938024Z", "start_time": "2023-11-09T14:48:31.759090Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -272,20 +267,20 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T14:49:06.568434Z", "start_time": "2023-11-09T14:49:06.468046Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ "class TextClassificationModel(nn.Module):\n", " def __init__(self, vocab_size, embed_dim, num_class):\n", " super(TextClassificationModel, self).__init__()\n", - " self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse_output=False)\n", + " self.embedding = nn.EmbeddingBag(vocab_size, embed_dim)\n", " self.fc = nn.Linear(embed_dim, num_class)\n", " self.init_weights()\n", "\n", @@ -436,23 +431,23 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Scan your model for vulnerabilities with Giskard\n", "\n", "Giskard's scan allows you to detect vulnerabilities in your model automatically. These include performance biases, unrobustness, data leakage, stochasticity, underconfidence, ethical issues, and more. For detailed information about the scan feature, please refer to our [scan documentation](https://docs.giskard.ai/en/stable/open_source/scan/scan_nlp/index.html)." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", @@ -469,16 +464,771 @@ "cell_type": "code", "execution_count": 11, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T15:20:30.991378Z", "start_time": "2023-11-09T15:20:30.768125Z" - } + }, + "collapsed": false }, "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -490,12 +1240,12 @@ }, { "cell_type": "markdown", - "source": [ - "## Generate comprehensive test suites automatically for your model" - ], "metadata": { "collapsed": false - } + }, + "source": [ + "## Generate comprehensive test suites automatically for your model" + ] }, { "cell_type": "markdown", @@ -512,28 +1262,202 @@ "cell_type": "code", "execution_count": 12, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T15:20:39.594648Z", "start_time": "2023-11-09T15:20:36.942910Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 14:00:13,219 pid:68530 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 14:00:13,225 pid:68530 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (7600, 2) executed in 0:00:00.016051\n", + "2024-05-29 14:00:13,689 pid:68530 MainThread giskard.datasets.base INFO Casting dataframe columns from {'text': 'object'} to {'text': 'object'}\n", + "2024-05-29 14:00:13,922 pid:68530 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (7600, 2) executed in 0:00:00.253734\n", + "2024-05-29 14:00:13,928 pid:68530 MainThread giskard.utils.logging_utils INFO Perturb and predict data executed in 0:00:00.729718\n", + "2024-05-29 14:00:13,930 pid:68530 MainThread giskard.utils.logging_utils INFO Compare and predict the data executed in 0:00:00.000818\n", + "Executed 'Invariance to “Add typos”' with arguments {'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", " Test failed\n", - " Metric: 0.89\n", - " - [TestMessageLevel.INFO] 7587 rows were perturbed\n", - " \n" + " Metric: 0.92\n", + " - [INFO] 7591 rows were perturbed\n", + " \n", + "2024-05-29 14:00:13,932 pid:68530 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 14:00:13,932 pid:68530 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 14:00:13,932 pid:68530 MainThread giskard.core.suite INFO Invariance to “Add typos” ({'model': , 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}): {failed, metric=0.917138716901594}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Invariance to “Add typos”\n
\n \n Measured Metric = 0.89073\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 94eabac3-e221-4832-b444-26794115dae9\n
\n \n
\n dataset\n Test Dataset\n
\n \n
\n transformation_function\n Add typos\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n
\n
\n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Invariance to “Add typos”\n", + "
\n", + " \n", + " Measured Metric = 0.91714\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Simple News Classification Model\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Test Dataset\n", + "
\n", + " \n", + "
\n", + " transformation_function\n", + " Add typos\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " output_sensitivity\n", + " 0.05\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + "
\n", + "\n" + ], + "text/plain": [ + "" + ] }, "execution_count": 12, "metadata": {}, @@ -574,118 +1498,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to: \n", - "\n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -708,7 +1520,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.10" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/tripadvisor_sentiment_classification.ipynb b/docs/reference/notebooks/tripadvisor_sentiment_classification.ipynb index 9e2cc242d3..177c145245 100644 --- a/docs/reference/notebooks/tripadvisor_sentiment_classification.ipynb +++ b/docs/reference/notebooks/tripadvisor_sentiment_classification.ipynb @@ -21,12 +21,7 @@ "Outline: \n", "\n", "* Detect vulnerabilities automatically with Giskard’s scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to: \n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -44,10 +39,10 @@ "execution_count": null, "id": "904bb40c24cd2d02", "metadata": { - "collapsed": false, "ExecuteTime": { "start_time": "2023-11-09T14:58:53.801518Z" - } + }, + "collapsed": false }, "outputs": [], "source": [ @@ -92,22 +87,30 @@ "from transformers import DistilBertForSequenceClassification, DistilBertTokenizer\n", "from typing import Union, List\n", "\n", - "from giskard import Dataset, Model, scan, Suite, GiskardClient, testing" + "from giskard import Dataset, Model, scan, testing" ] }, { "cell_type": "markdown", - "source": [ - "## Define constants" - ], + "id": "e25efcb2dd214fff", "metadata": { "collapsed": false }, - "id": "e25efcb2dd214fff" + "source": [ + "## Define constants" + ] }, { "cell_type": "code", "execution_count": 2, + "id": "9dc3ac4821372c97", + "metadata": { + "ExecuteTime": { + "end_time": "2023-11-09T15:22:42.596338Z", + "start_time": "2023-11-09T15:22:42.552639Z" + }, + "collapsed": false + }, "outputs": [], "source": [ "# Constants\n", @@ -123,19 +126,19 @@ "DATA_URL = \"ftp://sys.giskard.ai/pub/unit_test_resources/tripadvisor_reviews_dataset/{}\"\n", "DATA_PATH = Path.home() / \".giskard\" / \"tripadvisor_reviews_dataset\"\n", "DATA_FILE_NAME = \"tripadvisor_hotel_reviews.csv\"" - ], - "metadata": { - "collapsed": false, - "ExecuteTime": { - "end_time": "2023-11-09T15:22:42.596338Z", - "start_time": "2023-11-09T15:22:42.552639Z" - } - }, - "id": "9dc3ac4821372c97" + ] }, { "cell_type": "code", "execution_count": 3, + "id": "7d58054eb0568a57", + "metadata": { + "ExecuteTime": { + "end_time": "2023-11-09T15:22:46.542276Z", + "start_time": "2023-11-09T15:22:46.501005Z" + }, + "collapsed": false + }, "outputs": [], "source": [ "# Set random seeds\n", @@ -143,39 +146,35 @@ "np.random.seed(RANDOM_SEED)\n", "torch.manual_seed(RANDOM_SEED)\n", "torch.cuda.manual_seed_all(RANDOM_SEED)" - ], - "metadata": { - "collapsed": false, - "ExecuteTime": { - "end_time": "2023-11-09T15:22:46.542276Z", - "start_time": "2023-11-09T15:22:46.501005Z" - } - }, - "id": "7d58054eb0568a57" + ] }, { "cell_type": "markdown", - "source": [ - "## Dataset preparation" - ], + "id": "d77b1e9844a959b6", "metadata": { "collapsed": false }, - "id": "d77b1e9844a959b6" + "source": [ + "## Dataset preparation" + ] }, { "cell_type": "markdown", - "source": [ - "### Load data" - ], + "id": "dda9fcb9495b3477", "metadata": { "collapsed": false }, - "id": "dda9fcb9495b3477" + "source": [ + "### Load data" + ] }, { "cell_type": "code", "execution_count": null, + "id": "e3c3e6a5", + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "nltk.download('stopwords')\n", @@ -285,11 +284,7 @@ " df.drop(columns=\"Rating\", inplace=True)\n", " df = text_preprocessor(df)\n", " return df" - ], - "metadata": { - "collapsed": false - }, - "id": "e3c3e6a5" + ] }, { "attachments": {}, @@ -303,7 +298,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "id": "8223c2c2", "metadata": { "ExecuteTime": { @@ -451,7 +446,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "id": "931c5a5c", "metadata": {}, "outputs": [], @@ -468,25 +463,25 @@ }, { "cell_type": "markdown", - "source": [ - "## Detect vulnerabilities in your model" - ], + "id": "19aace313efe15af", "metadata": { "collapsed": false }, - "id": "19aace313efe15af" + "source": [ + "## Detect vulnerabilities in your model" + ] }, { "cell_type": "markdown", + "id": "cd3d8739ca1234a5", + "metadata": { + "collapsed": false + }, "source": [ "### Scan your model for vulnerabilities with Giskard\n", "\n", "Giskard's scan allows you to detect vulnerabilities in your model automatically. These include performance biases, unrobustness, data leakage, stochasticity, underconfidence, ethical issues, and more. For detailed information about the scan feature, please refer to our [scan documentation](https://docs.giskard.ai/en/stable/open_source/scan/scan_nlp/index.html)." - ], - "metadata": { - "collapsed": false - }, - "id": "cd3d8739ca1234a5" + ] }, { "cell_type": "code", @@ -500,7 +495,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 10, "id": "ecb49fa5", "metadata": { "ExecuteTime": { @@ -511,7 +506,2410 @@ "outputs": [ { "data": { - "text/html": "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -541,7 +2939,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 11, "id": "bea736a9", "metadata": { "ExecuteTime": { @@ -554,127 +2952,898 @@ "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Invariance to “Switch Religion”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", - " Test failed\n", - " Metric: 0.88\n", - " - [TestMessageLevel.INFO] 8 rows were perturbed\n", - " \n", - "Executed 'Invariance to “Switch countries from high- to low-income and vice versa”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", + "2024-05-29 14:07:23,236 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,237 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (54, 2) executed in 0:00:00.009851\n", + "Executed 'Precision on data slice “`Review` contains \"manager\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.85\n", - " - [TestMessageLevel.INFO] 137 rows were perturbed\n", + " Metric: 0.24\n", " \n", - "Executed 'Invariance to “Switch Gender”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", - " Test failed\n", - " Metric: 0.95\n", - " - [TestMessageLevel.INFO] 396 rows were perturbed\n", - " \n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/Users/mykytaalekseiev/Work/GiskardDevelopVersion/giskard-client/.venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py:59: FutureWarning: 'DataFrame.swapaxes' is deprecated and will be removed in a future version. Please use 'DataFrame.transpose' instead.\n", - " return bound(*args, **kwds)\n", - "/Users/mykytaalekseiev/Work/GiskardDevelopVersion/giskard-client/.venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2614: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n", - " warnings.warn(\n", - "/var/folders/4q/3_bfyqnn7yv5jcjq98x2jf680000gn/T/ipykernel_27805/3117328017.py:61: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n", - " probs = torch.nn.functional.softmax(outputs.logits).detach().cpu().numpy()\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Executed 'Invariance to “Add typos”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'transformation_function': , 'threshold': 0.95, 'output_sensitivity': 0.05}: \n", - " Test failed\n", - " Metric: 0.67\n", - " - [TestMessageLevel.INFO] 999 rows were perturbed\n", " \n", - "Executed 'Precision on data slice “`Review` contains \"loved\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,259 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,260 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (56, 2) executed in 0:00:00.007041\n", + "Executed 'Precision on data slice “`Review` contains \"tiny\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.07\n", + " Metric: 0.3\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"complimentary\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,278 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,279 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (74, 2) executed in 0:00:00.005584\n", + "Executed 'Precision on data slice “`Review` contains \"said\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.08\n", + " Metric: 0.39\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"wonderful\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,297 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,298 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (76, 2) executed in 0:00:00.005489\n", + "Executed 'Precision on data slice “`Review` contains \"air\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.09\n", + " Metric: 0.39\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"perfect\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,316 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,319 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (57, 2) executed in 0:00:00.009502\n", + "Executed 'Precision on data slice “`Review` contains \"elevator\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.1\n", + " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"quarter\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,337 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,338 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (52, 2) executed in 0:00:00.006870\n", + "Executed 'Precision on data slice “`Review` contains \"reservation\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.1\n", + " Metric: 0.4\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"excellent\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,355 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,356 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (93, 2) executed in 0:00:00.005766\n", + "Executed 'Precision on data slice “`Review` contains \"bad\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.1\n", + " Metric: 0.43\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"french\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,374 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,375 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (60, 2) executed in 0:00:00.006260\n", + "Executed 'Precision on data slice “`Review` contains \"hear\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.1\n", + " Metric: 0.43\n", " \n", " \n", - "Executed 'Precision on data slice “`avg_word_length(Review)` >= 5.983”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,392 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,393 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (53, 2) executed in 0:00:00.006900\n", + "Executed 'Precision on data slice “`Review` contains \"average\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.1\n", + " Metric: 0.43\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"fantastic\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,597 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,598 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (83, 2) executed in 0:00:00.006237\n", + "Executed 'Precision on data slice “`Review` contains \"work\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.11\n", + " Metric: 0.45\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"choice\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,615 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,616 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (61, 2) executed in 0:00:00.006405\n", + "Executed 'Precision on data slice “`Review` contains \"noisy\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.11\n", + " Metric: 0.48\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"beautiful\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,632 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,633 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (83, 2) executed in 0:00:00.006357\n", + "Executed 'Precision on data slice “`Review` contains \"times\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.11\n", + " Metric: 0.48\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"enjoyed\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,648 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,649 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (94, 2) executed in 0:00:00.006226\n", + "Executed 'Precision on data slice “`Review` contains \"days\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.11\n", + " Metric: 0.49\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"love\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,666 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,668 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (73, 2) executed in 0:00:00.006877\n", + "Executed 'Precision on data slice “`Review` contains \"know\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.12\n", + " Metric: 0.49\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"suite\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", + "2024-05-29 14:07:23,684 pid:70250 MainThread giskard.datasets.base INFO Casting dataframe columns from {'Review': 'object'} to {'Review': 'object'}\n", + "2024-05-29 14:07:23,685 pid:70250 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (81, 2) executed in 0:00:00.006585\n", + "Executed 'Precision on data slice “`Review` contains \"minutes\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}: \n", " Test failed\n", - " Metric: 0.12\n", + " Metric: 0.49\n", " \n", " \n", - "Executed 'Precision on data slice “`Review` contains \"orleans\"”' with arguments {'model': <__main__.GiskardModelCustomWrapper object at 0x12dd82590>, 'dataset': , 'slicing_function': , 'threshold': 0.19474999999999998}: \n", - " Test failed\n", - " Metric: 0.12\n", - " \n", - " \n" + "2024-05-29 14:07:23,688 pid:70250 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 14:07:23,688 pid:70250 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 14:07:23,688 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"manager\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.24074074074074073}\n", + "2024-05-29 14:07:23,688 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"tiny\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.30357142857142855}\n", + "2024-05-29 14:07:23,689 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"said\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.3918918918918919}\n", + "2024-05-29 14:07:23,689 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"air\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.39473684210526316}\n", + "2024-05-29 14:07:23,689 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"elevator\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.40350877192982454}\n", + "2024-05-29 14:07:23,689 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"reservation\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.40384615384615385}\n", + "2024-05-29 14:07:23,690 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"bad\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.43010752688172044}\n", + "2024-05-29 14:07:23,690 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"hear\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.43333333333333335}\n", + "2024-05-29 14:07:23,690 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"average\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.4339622641509434}\n", + "2024-05-29 14:07:23,690 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"work\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.4457831325301205}\n", + "2024-05-29 14:07:23,691 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"noisy\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.47540983606557374}\n", + "2024-05-29 14:07:23,691 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"times\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.4819277108433735}\n", + "2024-05-29 14:07:23,691 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"days\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.48936170212765956}\n", + "2024-05-29 14:07:23,691 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"know\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.4931506849315068}\n", + "2024-05-29 14:07:23,692 pid:70250 MainThread giskard.core.suite INFO Precision on data slice “`Review` contains \"minutes\"” ({'model': <__main__.GiskardModelCustomWrapper object at 0x3215bda50>, 'dataset': , 'slicing_function': , 'threshold': 0.61275}): {failed, metric=0.49382716049382713}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Invariance to “Switch Religion”\n
\n \n Measured Metric = 0.875\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n transformation_function\n Switch Religion\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Invariance to “Switch countries from high- to low-income and vice versa”\n
\n \n Measured Metric = 0.85401\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n transformation_function\n Switch countries from high- to low-income and vice versa\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Invariance to “Switch Gender”\n
\n \n Measured Metric = 0.94949\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n transformation_function\n Switch Gender\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n
\n \n \n
\n Test Invariance to “Add typos”\n
\n \n Measured Metric = 0.66567\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n transformation_function\n Add typos\n
\n \n
\n threshold\n 0.95\n
\n \n
\n output_sensitivity\n 0.05\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "loved"”\n
\n \n Measured Metric = 0.07407\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "loved"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "complimentary"”\n
\n \n Measured Metric = 0.07547\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "complimentary"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "wonderful"”\n
\n \n Measured Metric = 0.09434\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "wonderful"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "perfect"”\n
\n \n Measured Metric = 0.09677\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "perfect"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "quarter"”\n
\n \n Measured Metric = 0.09804\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "quarter"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "excellent"”\n
\n \n Measured Metric = 0.10317\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "excellent"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "french"”\n
\n \n Measured Metric = 0.10345\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "french"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`avg_word_length(Review)` >= 5.983”\n
\n \n Measured Metric = 0.10471\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `avg_word_length(Review)` >= 5.983\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "fantastic"”\n
\n \n Measured Metric = 0.10714\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "fantastic"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "choice"”\n
\n \n Measured Metric = 0.10909\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "choice"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "beautiful"”\n
\n \n Measured Metric = 0.10989\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "beautiful"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "enjoyed"”\n
\n \n Measured Metric = 0.1134\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "enjoyed"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "love"”\n
\n \n Measured Metric = 0.11594\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "love"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "suite"”\n
\n \n Measured Metric = 0.11688\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "suite"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n \n
\n Test Precision on data slice “`Review` contains "orleans"”\n
\n \n Measured Metric = 0.12069\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 42c0d917-a7d4-43b9-a4ca-85b3f9309554\n
\n \n
\n dataset\n Trip advisor reviews sentiment\n
\n \n
\n slicing_function\n `Review` contains "orleans"\n
\n \n
\n threshold\n 0.19474999999999998\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "manager"”\n", + "
\n", + " \n", + " Measured Metric = 0.24074\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "manager"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "tiny"”\n", + "
\n", + " \n", + " Measured Metric = 0.30357\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "tiny"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "said"”\n", + "
\n", + " \n", + " Measured Metric = 0.39189\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "said"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "air"”\n", + "
\n", + " \n", + " Measured Metric = 0.39474\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "air"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "elevator"”\n", + "
\n", + " \n", + " Measured Metric = 0.40351\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "elevator"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "reservation"”\n", + "
\n", + " \n", + " Measured Metric = 0.40385\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "reservation"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "bad"”\n", + "
\n", + " \n", + " Measured Metric = 0.43011\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "bad"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "hear"”\n", + "
\n", + " \n", + " Measured Metric = 0.43333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "hear"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "average"”\n", + "
\n", + " \n", + " Measured Metric = 0.43396\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "average"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "work"”\n", + "
\n", + " \n", + " Measured Metric = 0.44578\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "work"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "noisy"”\n", + "
\n", + " \n", + " Measured Metric = 0.47541\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "noisy"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "times"”\n", + "
\n", + " \n", + " Measured Metric = 0.48193\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "times"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "days"”\n", + "
\n", + " \n", + " Measured Metric = 0.48936\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "days"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "know"”\n", + "
\n", + " \n", + " Measured Metric = 0.49315\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "know"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Precision on data slice “`Review` contains "minutes"”\n", + "
\n", + " \n", + " Measured Metric = 0.49383\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " Trip advisor sentiment classifier\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " Trip advisor reviews sentiment\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `Review` contains "minutes"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.61275\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 15, + "execution_count": 11, "metadata": {}, "output_type": "execute_result" } @@ -711,122 +3880,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - }, - "id": "29b180b1bab598bf" - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - }, - "id": "45cc31e6aa286189" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - }, - "id": "4ac0c57148bca48a" - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - }, - "id": "ba2b6decf6ef3f9d" - }, - { - "cell_type": "markdown", - "id": "cf824254", - "metadata": {}, - "source": [ - "### Upload your test suite to the Giskard Hub\n", - "\n", - "The entry point to the Giskard Hub is the upload of your test suite. Uploading the test suite will automatically save the model, dataset, tests, slicing & transformation functions to the Giskard Hub." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "8efd6bf3", - "metadata": {}, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "id": "639f0c2d048805be", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - " \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - }, - "id": "30133c15995b7cb1" } ], "metadata": { @@ -845,7 +3898,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.11" + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/notebooks/twitter_sentiment_analysis_roberta.ipynb b/docs/reference/notebooks/twitter_sentiment_analysis_roberta.ipynb index 9ea92c4248..33d9682192 100644 --- a/docs/reference/notebooks/twitter_sentiment_analysis_roberta.ipynb +++ b/docs/reference/notebooks/twitter_sentiment_analysis_roberta.ipynb @@ -23,12 +23,7 @@ "Outline:\n", "\n", "* Detect vulnerabilities automatically with Giskard's scan\n", - "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics\n", - "* Upload your model to the Giskard Hub to:\n", - "\n", - " * Debug failing tests & diagnose issues\n", - " * Compare models & decide which one to promote\n", - " * Share your results & collect feedback from non-technical team members" + "* Automatically generate & curate a comprehensive test suite to test your model beyond accuracy-related metrics" ] }, { @@ -67,7 +62,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 1, "id": "initial_id", "metadata": { "ExecuteTime": { @@ -85,7 +80,7 @@ "from datasets import load_dataset\n", "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n", "\n", - "from giskard import Dataset, Model, scan, testing, GiskardClient, Suite " + "from giskard import Dataset, Model, scan, testing " ] }, { @@ -101,7 +96,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 2, "id": "856d520e800188b4", "metadata": { "ExecuteTime": { @@ -154,7 +149,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": null, "id": "50d7817e0425d02c", "metadata": { "ExecuteTime": { @@ -183,7 +178,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": null, "id": "9ff7255a655f3cc5", "metadata": { "ExecuteTime": { @@ -225,7 +220,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": null, "id": "e89a8d482a455064", "metadata": { "ExecuteTime": { @@ -254,7 +249,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": null, "id": "87c897d849e9e004", "metadata": { "ExecuteTime": { @@ -318,7 +313,7 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 8, "id": "7ae720b00159c8a6", "metadata": { "ExecuteTime": { @@ -332,7 +327,7 @@ { "data": { "text/html": [ - "\n" + "text/html": [ + "\n", + "" + ] }, "metadata": {}, "output_type": "display_data" @@ -433,98 +2824,794 @@ }, { "cell_type": "markdown", + "metadata": { + "collapsed": false + }, "source": [ "### Generate test suites from the scan\n", "\n", "The objects produced by the scan can be used as fixtures to generate a test suite that integrate all detected vulnerabilities. Test suites allow you to evaluate and validate your model's performance, ensuring that it behaves as expected on a set of predefined test cases, and to identify any regressions or issues that might arise during development or updates." - ], - "metadata": { - "collapsed": false - } + ] }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 12, "metadata": { - "collapsed": false, "ExecuteTime": { "end_time": "2023-11-09T16:42:10.035419Z", "start_time": "2023-11-09T16:42:09.270253Z" - } + }, + "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Executed 'Overconfidence on data slice “`hours-per-week` < 41.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5045638359329867, 'p_threshold': 0.5}: \n", + "2024-05-29 14:14:33,648 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,654 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (6902, 10) executed in 0:00:00.041066\n", + "Executed 'Overconfidence on data slice “`hours-per-week` < 41.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4973034997131383, 'p_threshold': 0.5}: \n", " Test failed\n", - " Metric: 0.51\n", + " Metric: 0.5\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`relationship` == \"Husband\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.01384993346299519, 'p_threshold': 0.95}: \n", + "2024-05-29 14:14:33,676 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,684 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (879, 10) executed in 0:00:00.017064\n", + "Executed 'Underconfidence on data slice “`age` >= 41.500 AND `age` < 45.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.02\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`age` >= 40.500 AND `age` < 56.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.01384993346299519, 'p_threshold': 0.95}: \n", + "2024-05-29 14:14:33,714 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,717 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (3923, 10) executed in 0:00:00.018021\n", + "Executed 'Underconfidence on data slice “`relationship` == \"Husband\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.02\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`fnlwgt` < 128385.000 AND `fnlwgt` >= 99990.000”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.01384993346299519, 'p_threshold': 0.95}: \n", + "2024-05-29 14:14:33,742 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,745 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1335, 10) executed in 0:00:00.017211\n", + "Executed 'Underconfidence on data slice “`age` >= 48.500 AND `age` < 58.500”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.02\n", " \n", " \n", - "Executed 'Underconfidence on data slice “`gender` == \"Male\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.01384993346299519, 'p_threshold': 0.95}: \n", + "2024-05-29 14:14:33,772 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,776 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (6485, 10) executed in 0:00:00.022294\n", + "Executed 'Underconfidence on data slice “`gender` == \"Male\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}: \n", " Test failed\n", " Metric: 0.01\n", " \n", " \n", - "Executed 'Recall on data slice “`relationship` == \"Own-child\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5277777777777778}: \n", + "2024-05-29 14:14:33,790 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,792 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1498, 10) executed in 0:00:00.010130\n", + "Executed 'Recall on data slice “`relationship` == \"Own-child\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}: \n", " Test failed\n", - " Metric: 0.26\n", + " Metric: 0.3\n", " \n", " \n", - "Executed 'Recall on data slice “`workclass` == \"?\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5277777777777778}: \n", + "2024-05-29 14:14:33,807 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,809 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (576, 10) executed in 0:00:00.007495\n", + "Executed 'Recall on data slice “`workclass` == \"?\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}: \n", " Test failed\n", - " Metric: 0.33\n", + " Metric: 0.34\n", " \n", " \n", - "Executed 'Recall on data slice “`relationship` == \"Not-in-family\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5277777777777778}: \n", + "2024-05-29 14:14:33,827 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,829 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (2528, 10) executed in 0:00:00.013633\n", + "Executed 'Recall on data slice “`relationship` == \"Not-in-family\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}: \n", " Test failed\n", - " Metric: 0.36\n", + " Metric: 0.35\n", " \n", " \n", - "Executed 'Recall on data slice “`workclass` == \"Self-emp-not-inc\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5277777777777778}: \n", + "2024-05-29 14:14:33,849 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,850 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (1076, 10) executed in 0:00:00.008265\n", + "Executed 'Recall on data slice “`relationship` == \"Unmarried\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}: \n", " Test failed\n", - " Metric: 0.39\n", + " Metric: 0.38\n", " \n", " \n", - "Executed 'Recall on data slice “`race` == \"Black\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5277777777777778}: \n", + "2024-05-29 14:14:33,865 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,867 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (935, 10) executed in 0:00:00.008937\n", + "Executed 'Recall on data slice “`race` == \"Black\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}: \n", " Test failed\n", - " Metric: 0.39\n", + " Metric: 0.38\n", " \n", " \n", - "Executed 'Recall on data slice “`relationship` == \"Unmarried\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5277777777777778}: \n", + "2024-05-29 14:14:33,880 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,882 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (721, 10) executed in 0:00:00.008114\n", + "Executed 'Recall on data slice “`workclass` == \"Self-emp-not-inc\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}: \n", " Test failed\n", - " Metric: 0.41\n", + " Metric: 0.39\n", " \n", " \n", - "Executed 'Recall on data slice “`gender` == \"Female\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5277777777777778}: \n", + "2024-05-29 14:14:33,902 pid:72955 MainThread giskard.datasets.base INFO Casting dataframe columns from {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'} to {'age': 'int64', 'workclass': 'object', 'fnlwgt': 'int64', 'relationship': 'object', 'race': 'object', 'gender': 'object', 'capital-gain': 'int64', 'capital-loss': 'int64', 'hours-per-week': 'int64'}\n", + "2024-05-29 14:14:33,905 pid:72955 MainThread giskard.utils.logging_utils INFO Predicted dataset with shape (3284, 10) executed in 0:00:00.014940\n", + "Executed 'Recall on data slice “`gender` == \"Female\"”' with arguments {'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}: \n", " Test failed\n", - " Metric: 0.5\n", + " Metric: 0.52\n", + " \n", " \n", - " \n" + "2024-05-29 14:14:33,917 pid:72955 MainThread giskard.core.suite INFO Executed test suite 'My first test suite'\n", + "2024-05-29 14:14:33,917 pid:72955 MainThread giskard.core.suite INFO result: failed\n", + "2024-05-29 14:14:33,917 pid:72955 MainThread giskard.core.suite INFO Overconfidence on data slice “`hours-per-week` < 41.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.4973034997131383, 'p_threshold': 0.5}): {failed, metric=0.5041237113402062}\n", + "2024-05-29 14:14:33,918 pid:72955 MainThread giskard.core.suite INFO Underconfidence on data slice “`age` >= 41.500 AND `age` < 45.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}): {failed, metric=0.023890784982935155}\n", + "2024-05-29 14:14:33,918 pid:72955 MainThread giskard.core.suite INFO Underconfidence on data slice “`relationship` == \"Husband\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}): {failed, metric=0.02013764975783839}\n", + "2024-05-29 14:14:33,918 pid:72955 MainThread giskard.core.suite INFO Underconfidence on data slice “`age` >= 48.500 AND `age` < 58.500” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}): {failed, metric=0.01647940074906367}\n", + "2024-05-29 14:14:33,918 pid:72955 MainThread giskard.core.suite INFO Underconfidence on data slice “`gender` == \"Male\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.011710512846760161, 'p_threshold': 0.95}): {failed, metric=0.013415574402467233}\n", + "2024-05-29 14:14:33,919 pid:72955 MainThread giskard.core.suite INFO Recall on data slice “`relationship` == \"Own-child\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}): {failed, metric=0.2962962962962963}\n", + "2024-05-29 14:14:33,919 pid:72955 MainThread giskard.core.suite INFO Recall on data slice “`workclass` == \"?\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}): {failed, metric=0.3448275862068966}\n", + "2024-05-29 14:14:33,919 pid:72955 MainThread giskard.core.suite INFO Recall on data slice “`relationship` == \"Not-in-family\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}): {failed, metric=0.3490909090909091}\n", + "2024-05-29 14:14:33,919 pid:72955 MainThread giskard.core.suite INFO Recall on data slice “`relationship` == \"Unmarried\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}): {failed, metric=0.38095238095238093}\n", + "2024-05-29 14:14:33,920 pid:72955 MainThread giskard.core.suite INFO Recall on data slice “`race` == \"Black\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}): {failed, metric=0.38333333333333336}\n", + "2024-05-29 14:14:33,920 pid:72955 MainThread giskard.core.suite INFO Recall on data slice “`workclass` == \"Self-emp-not-inc\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}): {failed, metric=0.391304347826087}\n", + "2024-05-29 14:14:33,920 pid:72955 MainThread giskard.core.suite INFO Recall on data slice “`gender` == \"Female\"” ({'model': , 'dataset': , 'slicing_function': , 'threshold': 0.5253512132822478}): {failed, metric=0.5231607629427792}\n" ] }, { "data": { - "text/plain": "", - "text/html": "\n\n\n\n\n\n
\n
\n
\n \n \n close\n \n \n Test suite failed.\n To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation)\n \n
\n
\n \n \n
\n Test Overconfidence on data slice “`hours-per-week` < 41.500”\n
\n \n Measured Metric = 0.50771\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `hours-per-week` < 41.500\n
\n \n
\n threshold\n 0.5045638359329867\n
\n \n
\n p_threshold\n 0.5\n
\n \n
\n
\n \n \n
\n Test Underconfidence on data slice “`relationship` == "Husband"”\n
\n \n Measured Metric = 0.02269\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `relationship` == "Husband"\n
\n \n
\n threshold\n 0.01384993346299519\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n
\n \n \n
\n Test Underconfidence on data slice “`age` >= 40.500 AND `age` < 56.500”\n
\n \n Measured Metric = 0.02032\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `age` >= 40.500 AND `age` < 56.500\n
\n \n
\n threshold\n 0.01384993346299519\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n
\n \n \n
\n Test Underconfidence on data slice “`fnlwgt` < 128385.000 AND `fnlwgt` >= 99990.000”\n
\n \n Measured Metric = 0.01793\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `fnlwgt` < 128385.000 AND `fnlwgt` >= 99990.000\n
\n \n
\n threshold\n 0.01384993346299519\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Underconfidence on data slice “`gender` == "Male"”\n
\n \n Measured Metric = 0.01496\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `gender` == "Male"\n
\n \n
\n threshold\n 0.01384993346299519\n
\n \n
\n p_threshold\n 0.95\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`relationship` == "Own-child"”\n
\n \n Measured Metric = 0.25926\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `relationship` == "Own-child"\n
\n \n
\n threshold\n 0.5277777777777778\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`workclass` == "?"”\n
\n \n Measured Metric = 0.32759\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `workclass` == "?"\n
\n \n
\n threshold\n 0.5277777777777778\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`relationship` == "Not-in-family"”\n
\n \n Measured Metric = 0.36364\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `relationship` == "Not-in-family"\n
\n \n
\n threshold\n 0.5277777777777778\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`workclass` == "Self-emp-not-inc"”\n
\n \n Measured Metric = 0.3913\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `workclass` == "Self-emp-not-inc"\n
\n \n
\n threshold\n 0.5277777777777778\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`race` == "Black"”\n
\n \n Measured Metric = 0.39167\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `race` == "Black"\n
\n \n
\n threshold\n 0.5277777777777778\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`relationship` == "Unmarried"”\n
\n \n Measured Metric = 0.4127\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `relationship` == "Unmarried"\n
\n \n
\n threshold\n 0.5277777777777778\n
\n \n
\n \n \n \n
\n Test Recall on data slice “`gender` == "Female"”\n
\n \n Measured Metric = 0.50409\n \n \n \n close\n \n \n Failed\n \n \n
\n
\n \n
\n model\n 018699ef-a664-48b8-b9a2-2db999094be4\n
\n \n
\n dataset\n salary_data\n
\n \n
\n slicing_function\n `gender` == "Female"\n
\n \n
\n threshold\n 0.5277777777777778\n
\n \n
\n \n \n\n \n\n" + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
\n", + "
\n", + "
\n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Test suite failed.\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Overconfidence on data slice “`hours-per-week` < 41.500”\n", + "
\n", + " \n", + " Measured Metric = 0.50412\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `hours-per-week` < 41.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.4973034997131383\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.5\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`age` >= 41.500 AND `age` < 45.500”\n", + "
\n", + " \n", + " Measured Metric = 0.02389\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `age` >= 41.500 AND `age` < 45.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.011710512846760161\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`relationship` == "Husband"”\n", + "
\n", + " \n", + " Measured Metric = 0.02014\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `relationship` == "Husband"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.011710512846760161\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`age` >= 48.500 AND `age` < 58.500”\n", + "
\n", + " \n", + " Measured Metric = 0.01648\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `age` >= 48.500 AND `age` < 58.500\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.011710512846760161\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Underconfidence on data slice “`gender` == "Male"”\n", + "
\n", + " \n", + " Measured Metric = 0.01342\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `gender` == "Male"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.011710512846760161\n", + "
\n", + " \n", + "
\n", + " p_threshold\n", + " 0.95\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`relationship` == "Own-child"”\n", + "
\n", + " \n", + " Measured Metric = 0.2963\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `relationship` == "Own-child"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5253512132822478\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`workclass` == "?"”\n", + "
\n", + " \n", + " Measured Metric = 0.34483\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `workclass` == "?"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5253512132822478\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`relationship` == "Not-in-family"”\n", + "
\n", + " \n", + " Measured Metric = 0.34909\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `relationship` == "Not-in-family"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5253512132822478\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`relationship` == "Unmarried"”\n", + "
\n", + " \n", + " Measured Metric = 0.38095\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `relationship` == "Unmarried"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5253512132822478\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`race` == "Black"”\n", + "
\n", + " \n", + " Measured Metric = 0.38333\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `race` == "Black"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5253512132822478\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`workclass` == "Self-emp-not-inc"”\n", + "
\n", + " \n", + " Measured Metric = 0.3913\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `workclass` == "Self-emp-not-inc"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5253512132822478\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " Test Recall on data slice “`gender` == "Female"”\n", + "
\n", + " \n", + " Measured Metric = 0.52316\n", + " \n", + " \n", + " \n", + " close\n", + " \n", + " \n", + " Failed\n", + " \n", + " \n", + "
\n", + "
\n", + " \n", + "
\n", + " model\n", + " salary_cls\n", + "
\n", + " \n", + "
\n", + " dataset\n", + " salary_data\n", + "
\n", + " \n", + "
\n", + " slicing_function\n", + " `gender` == "Female"\n", + "
\n", + " \n", + "
\n", + " threshold\n", + " 0.5253512132822478\n", + "
\n", + " \n", + "
\n", + " \n", + " \n", + "\n", + " \n", + "\n" + ], + "text/plain": [ + "" + ] }, - "execution_count": 14, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -563,107 +3650,6 @@ "source": [ "test_suite.add_test(testing.test_f1(model=giskard_model, dataset=giskard_dataset, threshold=0.7)).run()" ] - }, - { - "cell_type": "markdown", - "source": [ - "## Debug and interact with your tests in the Giskard Hub\n", - "\n", - "At this point, you've created a test suite that is highly specific to your domain & use-case. Failing tests can be a pain to debug, which is why we encourage you to head over to the Giskard Hub.\n", - "\n", - "Play around with a demo of the Giskard Hub on HuggingFace Spaces using [this link](https://huggingface.co/spaces/giskardai/giskard).\n", - "\n", - "More than just debugging tests, the Giskard Hub allows you to:\n", - "\n", - "* Compare models to decide which model to promote\n", - "* Automatically create additional domain-specific tests through our automated model insights feature\n", - "* Share your test results with team members and decision makers\n", - "\n", - "The Giskard Hub can be deployed easily on HuggingFace Spaces." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "Here's a sneak peek of automated model insights on a credit scoring classification model." - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.09.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABR4AAAGUCAYAAABTOz7qAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFHqADAAQAAAABAAABlAAAAABBU0NJSQAAAFNjcmVlbnNob3QWVvFBAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj40MDQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxMDwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgp7E0vvAABAAElEQVR4AezdB3xUVfrw8WdCCAECJPQO0gQEkaJUlSaoWBBwFRV7ef1bV9a2lrUs9rKrqGvHXlCQqiAgSu+9dwi9BQhJIGXe+5xwL3cmkwaTMJP8zuczzC3nnnvu97Lu4ZlTPF4rCQkBBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiiQEQQy6IoBBBAAAEEEEAAAQQQQAABBBBAAAEEEEDACBB45C8CAggggAACCCCAAAIIIIAAAggggAACCARdgMBj0EkpEAEEEEAAAQQQQAABBBBAAAEEEEAAAQQIPPJ3AAEEEEAAAQQQQAABBBBAAAEEEEAAAQSCLkDgMeikFIgAAggggAACCCCAAAIIIIAAAggggAACBB75O4AAAggggAACCCCAAAIIIIAAAggggAACQRcg8Bh0UgpEAAEEEEAAAQQQQAABBBBAAAEEEEAAAQKP/B1AAAEEEEAAAQQQQAABBBBAAAEEEEAAgaALEHgMOikFIoAAAggggAACCCCAAAIIIIAAAggggACBR/4OIIAAAggggAACCCCAAAIIIIAAAggggEDQBQg8Bp2UAhFAAAEEEEAAAQQQQAABBBBAAAEEEECAwCN/BxBAAAEEEEAAAQQQQAABBBBAAAEEEEAg6AIEHoNOSoEIIIAAAggggAACCCCAAAIIIIAAAgggQOCRvwMIIIAAAggggAACCCCAAAIIIIAAAgggEHQBAo9BJ6VABBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyN8BBBBAAAEEEEAAAQQQQAABBBBAAAEEEAi6AIHHoJNSIAIIIIAAAggggAACCCCAAAIIIIAAAggQeOTvAAIIIIAAAggggAACCCCAAAIIIIAAAggEXSDyVEtMSUmR5ORkSU1NPdUiuA4BBBBAAAEEEEAAAQQQQAABBBBAAAEEwkAgKipKSpcuLaVKlcpzbT1eK+U5t5VRs+/bt0+SkpLE4/GI3lS/SQgggAACCCCAAAIIIIAAAggggAACCCBQ9AQyMjJM50ONC5YrV04qVqyYp4fMd4/HvXv3ivZ21BvojUgIIIAAAggggAACCCCAAAIIIIAAAgggUPQFEhIS5NChQ6KByMqVK+f6wPkKPB45csQMr9aCy5Ytm2vhZEAAAQQQQAABBBBAAAEEEEAAAQQQQACBoiEQGxtrRj5rAFKHXecWH8zX4jIaeNRx3LkVWjQoeQoEEEAAAQQQQAABBBBAAAEEEEAAAQQQcAtUqFBBIiMjJTEx0X044HaeA4/2WO7o6OiABXEQAQQQQAABBBBAAAEEEEAAAQQQQAABBIq+gHZMPH78eK4PmufAo70GjUY0SQgggAACCCCAAAIIIIAAAggggAACCCBQPAU0PmjHCnMSyHPgMadCOIcAAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUIPLo12EYAAQQQQAABBBBAAAEEEEAAAQQQQACBoAgQeAwKI4UggAACCCCAAAIIIIAAAggggAACCCCAgFuAwKNbg20EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIoAgcegMFIIAggggAACCCCAAAIIIIAAAggggAACCLgFCDy6NdhGAAEEEEAAAQQQQAABBBBAAAEEEEAAgaAIEHgMCiOFIIAAAggggAACCCCAAAIIIIAAAggggIBbgMCjW4NtBBBAAAEEEEAAAQQQQAABBBBAAAEEEAiKAIHHoDBSCAIIIIAAAggggAACCCCAAAIIIIAAAgi4BQg8ujXYRgABBBBAAAEEEEAAAQQQQAABBBBAAIGgCBB4DAojhSCAAAIIIIAAAggggAACCCCAAAIIIICAW4DAo1uDbQQQQAABBBBAAAEEEEAAAQQQQAABBBAIigCBx6AwUggCCCCAAAIIIIAAAggggAACCCCAAAIIuAUi3TtsI5BvgX1rxbt/vcjhHSLJCSLpx0UirL9W0eVFYqqLp2IDkerniHhK5LtoLkAAAQQQQAABBBBAAAEEEEAAAQQQCF8BAo/h++7OWM29ayeIbJgs3i2zRI4dPlmP6AoikaWs4GOadfyQSEa6ePVsRAnx1Okg0rCreM6+XCQq5uQ1bCGAAAIIIIAAAggggAACCCCAAAIIFEkBAo9F8rUWwEOlp4p3wTDxLv1B5OhekQq1xNPUCiLWbCOeKmeLxNbJ2qvxyE7x7lsnsmOxeLfOFJkyRLxTXxXPuX8TT5ubRcrVKICKUiQCCCCAAAIIIIAAAggggAACCCCAQCgIeLxWyktF0tPTJT4+XipVqiQxMfRYy4tZUcnjXfGLeGcNNQFHT+NeIi36iadux/w/3p5V4l0xMjN4aV3t6fB/4ml/T/7L4QoEEEAAAQQQQAABBBBAAAEEEEAAgTMmkJCQIIcPH5a6devmWAd6PObIU8xPpiaJd/IL4l3zq3jqdRJPn7dEapx76ihVm4lHP21vFe/cD8U7+33TEzKi+zMilRqderlciQACCCCAAAIIIIAAAggggAACCCAQcgKsah1yryREKrR/g2T8eHNm0PHix8TT94PTCzq6H6t8TfH0fF48V7wtkhAvGT8MEu/Gqe4cbCOAAAIIIIAAAggggAACCCCAAAIIhLkAgccwf4EFUv29qyVjpDUE2lqlOqL/p+I578YCuY2nYXeJuP5rEWuOSO+Yh8QsWlMgd6JQBBBAAAEEEEAAAQQQQAABBBBAAIHCFiDwWNjioX6/wzskY+zfRUqUlIhrPhSp3a5ga2wtMBPR7yMzZ6T318eslbJnFOz9KB0BBBBAAAEEEEAAAQQQQAABBBBAoFAECDwWCnP43CRj4tMiKYckQodBV2pYOBUvESWeK/8jUq2FeCc8JWIFP0kIIIAAAggggAACCCCAAAIIIIAAAuEtQOAxDN5fUlKSLF261PkcOXKkQGrtnW4FG7cvEM8lL1rDn5uewj10gfQ8LZKetezIaIno9W+RtBTx/vFS1vMcQQABBBBAAAEEEEAAAQQQQAABBBAIK4GQXdVag23ZpYiICImOjs7udJE7vnr1ahkwYIDzXJ999pl07drV2Q/Kxs7F4l0wTDytB4mnUY/ci/RaAUZvmkhGuvWdYX2nWt8nAo8e669VRAkRj/XRb/2IJ/cyK54lnoufEO+kf4ksGy6eltfmfg05EEAAAQQQQAABBBBAAAEEEEAAAQRCUiBkA4+XX365bN26NVu02NhYqV27ttStW1cGDhwonTt3zjYvJ3IX8M75SKRcdfF0eSj3zBpkPGb1ujy6V7zWR44dtgKPVhBS07KfRJr3FSlVVjzRsSJlq4qUqSxSssyJAGRmtuz+9JxjXbthknjnfCiec/rl6ZrsyuI4AggggAACCCCAAAIIIIAAAggggMCZEwjZwGNuJAkJCaKf5cuXy/jx4+WCCy6Q1157zQQic7u2KJ7/5ZdfZP/+/ebR6tSpI7169crzY3q3zTGLuni6/dMK9JXM/jrt0WgNhZZDW8WbsCVz25Xbu2uZyI4l4tFAY8Ou4k3cLWJ9TAAy7qzMIGRklHVFzr0fPe3uFO/wW8S7+BvxtLnZdQc2EUAAAQQQQAABBBBAAAEEEEAAAQTCRSAsAo+RkZFSsWJFxzQjI0MOHDgg+m2nuXPnyr333isjRoyQUqVK2YeLzfcnn3wiK1euNM/bvXv3fAUeZcVIKyhYWTznXpeDlxV0PHZIZP/azF6OOsTanbQX5JoJ5oh341/iqdVGJLp85n5KgsjuZeKJqy8Sa31KlraO5xB8rHmeeOp3Ee/ynwk8WlIkBBBAAAEEEEAAAQQQQAABBBBAIBwFwmJxmaZNm8rs2bOdjwYZV6xYIV988YU0btzYcV+1apW8/ba1QAop7wLHE8W75lfxNL0y52uOHxXZZwUdE/dkzuvon3vjNLMatjlsgpC/+eawhmJ7D2wwvSUl7bjvuUB7zaz6HNws3vh5gc5yDAEEEEAAAQQQQAABBBBAAAEEEEAgxAXCIvAYyFB7NV544YUm+BgTE+Nk+euvv5xtNnIX8G6yAoZWynFBGa/Vu9EaWu1NtoZy60Iy/snq0ejdfKKcivXNWTPs+uAW35zWUG1vgjVvZ/KBwOW4cnsa9RQpYQ3LPlE/1yk2EUAAAQQQQAABBBBAAAEEEEAAAQTCQCAshlrn5Fi9enW56KKLzDyPmm/9+vWSlpYmOjzbP3mtwNeff/4pw4cPlw0bNkhiYqI0adJEtEdl3759zbb/Ne793bt3i64orUOat2zZIiVLlpR69epJixYt5LbbbpO4uDh3drM9duxYmTx5stkuUaKEvPHGG1nyaD2eeeYZ53j//v2lS5cuzn52G3PmzJHvv//enN6+fbuTTev397//3exfeeWVokOvs03b51uLv1jD2Ku3zDaLHN1jDa+2ejqmW8OpAyUdYp1uLS5T2nr+treIzP/S6q24Sbyrx4unw/+zopquYdU6R+ThbSJRVrC4VLlApWUei4gUT50LTI9H19XZ5+cMAggggAACCCCAAAIIIIAAAggggEBICWSNzoVU9fJWmRo1ajgZNei4b98+0YCkOx08eNDMAanDtN1px44dMnXqVPn888/lqaeekptuusl92tnWgKMGDVNSrMCZK23atMm5/sknn5QbbrjBdVZMkHLUqFHmWHaBx2PHjomdRzO2bt06T4FHXfXbfZ194127djnHNaiaU+DRu3uleKrlEHTUBWV05erUZLt4328NMO5annns7EutxWmsgGGzy8Q78wMrwLhDvNsXiqd2W59rvEf3iad8ohV41J6qOYQVNRg62ypHV8y2yiUhgAACCCCAAAIIIIAAAggggAACCISPQET4VDX7mm7evNk5qT0dq1Sp4uzrRlJSkgwYMEDcQceIiAgpV+5kjzsN/j377LNm6LbPxdaO9lr897//7RN0bNasmVSrVs3JevToUXO9BjELK2kgs3Tp0uajz2Mn93HtlZljsuZRlIoNss+Sas3teOxIZvAvSy5r6PSqX81Rj1WGp1rzzBzlali9Fdtlbq//3QpaHvO9UgOJuuBMmt9x31wn66UraJMQQAABBBBAAAEEEEAAAQQQQAABBMJK4GS0KqyqfbKyy5YtM8On7SPaW1ADb+708ssvi/ZM1KQBusGDB8vSpUtlyZIlMmXKFGnX7kSQzDo/dOhQE6g0mU/88Z///MfZ1aHVutDNuHHjZObMmfLdd985q2jrKtuvvvqqk7egN/r162cW2dGFdrRno50uvvhi5/jtt99uH876rQHF1CSRcicDqFkyWYFHbzZDrL3bFogc2WldYvVabHZ5ZjDx4Farh+Q+EZ2jMdJavfqYFbjcOCVLsaKL1egiNDkkT7nMXqveRKvHJQkBBBBAAAEEEEAAAQQQQAABBBBAIKwEwiLwqHMzaq9F+3PkyBFZs2aNfPzxxzJo0CAzp6Otfu+999qb5luHXdvzIOoBHXZ83333SZkyZcz5+vXry/vvvy9ly5Y1+/v375dvvvnGbOsfhw4dko0bNzr7Opdj1apVzb7Hmruwffv2cs011zjn161blyVw6ZwMtQ0N/mnS+RazS+nWCtS6uIx/0rkatTejlTx124vEWMHL+Pninfuxddya0zLK8m3c3Zz3bpmdGYw0eyf+0HKtQG2OKepEj1S7njlm5iQCCCCAAAIIIIAAAggggAACCCCAQCgJhMXEedqjTxdwyS09/PDD0rVrV59s48ePl/T0k4Ez/zkYNXPlypWlc+fOMnHiRHPt8uUn5iy09nTFbF00RueI1PT777+bYKcGHe30wgsvyD//+U97V6Kjo53tkN7Q+Rs1eXKIP5/IkpnR9ed6qxfjcau3ZEkrwNiom+vEyU2zOMy2eSKJe0RWW0Oy2w46eVK04OwKP5HNNg60krarJDYRQAABBBBAAAEEEEAAAQQQQAABBEJPICwCj7mx1apVS5544gnp06dPlqw6pNpOdi9HXdnaP7lXwd62zVp1+UTSYds9evSQn376yRyZMWOG9O7dW3S1aA1WtmrVyqygrQHKsEslTwRIs1s4Rh9IF3XxD0xai814t87JfFzt1ajBx0BJr2t6ubXK9TDx7lsrsneteKo0ycwZYc096V+ufxl2vUpaQ7ZJCCCAAAIIIIAAAggggAACCCCAAAJhJRAWgUcNCmqvRDvpXIp79li96E6kK664ImDQUU/v3r3bzmaGQOc45+GJnO7Aox56+umnzUrZ9sIx69evl7ffftt8NODYpUsXuf766+Wiiy5y7hUWG6XjrOCfNR+mrlqdXdKgov+K0qvGW50VrWHS1hyM2qsxp+Sp1FCkanPx7lkpssa6rnKjzICjBhM1+JhD8p6ol6dMxRxycQoBBBBAAAEEEEAAAQQQQAABBBBAIBQFchhjGzrV1YVTdCEX+zNr1iyfodfDhg2TXbt2Baywrjad3+R/Tfny5eWTTz6Rp556ymcRFy03MTFRfvvtN7n11lvl8ccf95lvMr/3PSP5Y+uI5LRqdKkY8egiMSeGPXv3rBbv/vWZVdXejLqwjJ1qtRPPBXeINLR6QbrT2b0zg5dH94t3y8zMwGOp8iIloty5sm7b9apg1ZGEAAIIIIAAAggggAACCCCAAAIIIBBWAmHR49FfVOdXfOSRR8TuvXjs2DHT+zDQitJ16tSRxYsXmyJ0SPaLL77oX1yWfV352j/psTvuuMN8tm7dauaDnD59umgQNDU1c3Xm4cOHy/Hjx01d/K8P1X1PlaZWb8RV7vChb1W1R2TZKpkrVusq2Gt+M+c91c4RqXiWb97SsSL68U9Wj0VP/U7i3fiXyIap4mnQVSS6ghWMtMrOKe22eklqYLTUiUVmcsrLOQQQQAABBBBAAAEEEEAAAQQQQACBkBIIy8CjCuoiMm3btpUFCxYY0BEjRsidd94pjRs39gF272uvSF2FunTp05szsG7duuZeej8d8n3jjTfKhg0bzH0nTJhggo9RUZm9+dxzR+rq3BokLVWqlE8dz+hOzdYia61g4qF4kQq1A1elbFXxHN0tGWsniCTtzxwiffalWfJ6dQ7H/evEW66GeGq18T3f4GKR7YtErOCld93v4mnY0/d8gD3v9vnWitkdA5zhEAIIIIAAAggggAACCCCAAAIIIIBAqAtk7doX6jV21e8f//iHs6crV7/xxhvOvr3RrFkze9Osbv3ll186+/4bupp1UlKSz+EhQ4ZI69atzUeDncnJyT7nq1atKg8++KBzLCUlRVavXu3s16xZ09nWuSnnzp3r7NsbBw4csDdP+du9yrZ7Fe/cCvTU62SyeDdOzT6rzscYZQ2N1h6LVvLU7xywZ6Pn0FZrKPVs8exbZ/L5/KHDqk8EK71Wr0exh2v7ZHLtbLcCyroadj3rXiQEEEAAAQQQQAABBBBAAAEEEEAAgbATCOvAo/ZevPDCCx3033//3ekBaR/s1q2bnHfeefauGQY9btw4Z9/eGDVqlAwYMEBuvvlmOXz4sH3Y9KA8dOiQ6EeHWH/77bfOOXvD7nWp+xoArF+/vn1KtHekO+l9tNejnbTH5N13323vnvJ3xYonF2BZtmyZaO/KPKVYq3612opXF37JIXkXfSOSlpIZcGxk9V4MkLwx1cRT/RzxBpqT0RpWHdHkEpFqLczCNBl/vhqghJOHvKutdxRVVjyNcu8ZefIqthBAAAEEEEAAAQQQQAABBBBAAAEEQkUgbIda24CDBw+WadOm2bvyyiuviM61aCedm/Gll16SK6+80vR41DkYH3jgAbNYTIsWLUyQcdGiRRIfbw01ttLChQtlzJgxZvi07vft21c++OAD2bJli+7Kyy+/LL/++qtoQLNEiRIye/ZsmTFjhjmnf3To0EF0MRo76XDw6tWrO4vf6JDwKVOmmBWwN27caHpHpqWl2dlP+btevXqOg/agvPjii6VVq1bSqVMnGThwYI7leppfLd7fnxXv5uniqd8la95dy8W7aow57mnYQzwpR8WbmiiSkW4FEU8GOD26YIw1NNosN3Mw00s8VmzbCjp6SluB0VJx4rF6PXp3LxfZsdgEOz1n6wI1funoHvEu/1k8592Y+zyQfpeyiwACCCCAAAIIIIAAAggggAACCCAQGgJhH3g899xzpVevXmaxFyXV3oeTJk2Snj1P9pTTVbGHDh1qVqW2hzUvWbJE9ONOGkjUxWeuv/5657DO1ajBwvvuu88EGXW4tAYn9eOftHfju+++63NY53N89NFHRQOkdkpISJDRo0fbu2al7PXr15/Witg33XST6Y2p9dOkgVT91KhRw7lPdhsm8Dj/MwtvmEiWwKNXMnsnZgYYNSB4MtSYXYlZjwe6xjv9P+Jp0E1Eh3K7knf+52bP0yrngKnrEjYRQAABBBBAAAEEEEAAAQQQQAABBEJMIKyHWtuWusK1eyXq119/3T7lfPfu3Vt0KHb//v2lcuXKznHd0B6Jl112mQwbNswn6GhniouLk6+++sr0lGzZsqWULFnSPmW+dZizzjc5duxYcQ95tjNdc801poel3sedypYta4Z2jxw50qeXpDtPXrebNGkib775plSoYK0W7UruuR9dh7Nses6/Q7zx80xPQ/dJM+R511L3oeBtJ+4WrwY83WnnEvEu/lY8bW7OXNHafY5tBBBAAAEEEEAAAQQQQAABBBBAAIGwEfBYcwEG6oyW5QF0wRLtQVepUiWJiYnJcj7cDmjPRx3qXLt2bRN4zE/9U1NTZc2aNaaHol7vH8jMqSyd03Hz5s2mJ6Jem9fAYE5lus/pUHItX3tVag/MatWseReteSfzkrxjHhTvtrkSMfAHkbh6ebkk6HkyvreGV6cclIhBI0VKhNDq30F/UgpEAAEEEEAAAQQQQAABBBBAAAEEwlNA4066Ror/2ib+T1NsA4/+EOxbAoe3S8Z31vDmig0k4lod7py3gGWw7LyTnhfvihHiuWqoeM46uWhQsMqnHAQQQAABBBBAAAEEEEAAAQQQQACB0xfIa+CxSAy1Pn0uSjAC5WuJ55IXrIVfFol33Mk5KQtDxzvjncygY5eHCToWBjj3QAABBBBAAAEEEEAAAQQQQAABBApYgMBjAQOHW/GeBl3F0/M58a6fLN4xD4ukHy/wR/BOe9Oa6/FT8bS9zXwK/IbcAAEEEEAAAQQQQAABBBBAAAEEEECgwAUIPBY4cfjdwHPONabno3fjH5Lxo7XIy+6VBfMQyQetnpWPiHfhl+K54C7xWL0dSQgggAACCCCAAAIIIIAAAggggAACRUOAwGPReI9BfwpP86vFc82HIskJkvH9QPHO+ySo9/CuGi0ZX/c3PSs9PZ4VT8f7g1o+hSGAAAIIIIAAAggggAACCCCAAAIInFmByDN7e+4eygKeuh3Ec8OP4p32hnhnvivelaPF02aQeFr0t9adObWYtRnCvfgbke0LxJR/4Qcilc8OZQbqhgACCCCAAAIIIIAAAggggAACCCBwCgKsan0KaMXxEu/WWSLzPxfvtjkipcqJp0lvkfoXiqdWW7OfrUlGmnitxWpkywzxrpskcmibSKVG1lyOt4in2VXZXsYJBBBAAAEEEEAAAQQQQAABBBBAAIHQFMjrqtb0eAzN9xdytfLU7ShifTxWT0XvqrFWEHGiyLKfxKs1ja0rngq1RUrHiZSIErGCjZJySLyHd4jsX5/5LFYPSU/DbiJd/i6eRj1C7vmoEAIIIIAAAggggAACCCCAAAIIIIBAcAXo8Rhcz+JV2vaF4t21RGTf+swgY8pBkTRrFewSVjy7VHnxlKshUrGBSPUW4ql9gUhkqeLlw9MigAACCCCAAAIIIIAAAggggAACRVCAHo9F8KWG3CPVamMNtW5jquUJucpRIQQQQAABBBBAAAEEEEAAAQQQQACBMylwaiuEnMkac28EEEAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAggggAACCCCAAAIIIIAAAggggAAC4SdA4DH83hk1RgABBBBAAAEEEEAAAQQQQAABBBBAIOQFCDyG/CuigggggAACCCCAAAIIIIAAAggggAACCISfAIHH8Htn1BgBBBBAAAEEEEAAAQQQQAABBBBAAIGQFyDwGPKviAoigAACCCCAAAIIIIAAAggggAACCCAQfgIEHsPvnVFjBBBAAAEEEEAAAQQQQAABBBBAAAEEQl6AwGPIvyIqiAACCCCAAAIIIIAAAggggAACCCCAQPgJEHgMv3dGjRFAAAEEEEAAAQQQQAABBBBAAAEEEAh5AQKPIf+KqCACCCCAAAIIIIAAAggggAACCCCAAALhJ0DgMfzeGTVGAAEEEEAAAQQQQAABBBBAAAEEEEAg5AUIPIb8K6KCCCCAAAIIIIAAAggggAACCCCAAAIIhJ8Agcfwe2fUGAEEEEAAAQQQQAABBBBAAAEEEEAAgZAXIPAY8q+ICiKAAAIIIIAAAggggAACCCCAAAIIIBB+AgQew++dUWMEEEAAAQQQQAABBBBAAAEEEEAAAQRCXoDAY8i/IiqIAAIIIIAAAggggAACCCCAAAIIIIBA+AkQeAy/d0aNEUAAAQQQQAABBBBAAAEEEEAAAQQQCHkBAo8h/4qoIAIIIIAAAggggAACCCCAAAIIIIAAAuEnQOAx/N4ZNUYAAQQQQAABBBBAAAEEEEAAAQQQQCDkBQg8hvwrooIIIIAAAggggAACCCCAAAIIIIAAAgiEnwCBx/B7Z9QYAQQQQAABBBBAAAEEEEAAAQQQQACBkBcg8Bjyr4gKIoAAAggggAACCCCAAAIIIIAAAgggEH4CBB7D751RYwQQQAABBBBAAAEEEEAAAQQQQAABBEJegMBjyL8iKogAAggggAACCCCAAAIIIIAAAggggED4CRB4DL93Ro0RQAABBBBAAAEEEEAAAQQQQAABBBAIeQECjyH/iqggAgggULQFjh07Jj///LNMmTKlQB504sSJpvz09PRTLn/27NmmjIMHD55yGVyIAAIIIIAAAgggEByBMWPGyKhRo4JTGKUggECBCkQWaOkUjgACCCAQ8gJpaWny7rvvZqlnZGSkVKlSRWrXri0dOnQQ3S+IdPz4cdHgYP369aV79+5Bv8XMmTNl586dctVVV0mJEiVyLf+DDz6QlJQUefjhh8Xj8Zj8S5YskYULF0q7du0kLi7OHJswYYJoQPLee++VqlWr5lruqlWr5Pvvv5e+fftK69atc81PBgQQQAABBBBA4EwKnOk2Yk7PPnXqVElNTZWrr77ayRaoDeecPM2Ngiz7NKvG5QiEvEDB/Csy5B+bCiKAAAII2AIZGRmyevVqiYiIkIoVK9qHRQOCy5cvN/u//fab3HPPPVKnTh3nfFHcUIsNGzaYhqw+f6lSpbJ9zM2bN5uAZkJCgk/gcfjw4aK9K6+//nqfa3ft2iX62bFjB4FHHxl2EEAAAQQQQCAUBcKpjZifNlx21ocOHRLtSVmrVi3p1q2bky0YZTuFsYFAMRQg8FgMXzqPjAACCAQSiImJkSFDhvic2rhxo8yYMUOmT58uH330kTz99NM5BuN8Lg7DHQ2+/vvf/xZtYOYUdNRHu+uuu+Tw4cMSGxvr86Rz5swxQVv/wKM2YM877zynx6TPRewggAACCCCAAAIhKhAObcT8tOGyY05KSpJp06ZJy5YtfQKPwSg7u3tyHIHiIMAcj8XhLfOMCCCAwCkKNGjQQAYNGmQaYHv27JGVK1cGLMnr9Yr2/NNv/6THNECXn3T06FHRuR/zkrSRqPnzmnRYTk75o6OjpUyZMrkWp41Q/6BjbhfZw7Szy6f10mFNuSUNjOp8k/osJAQQQAABBBBAoLAFgtFG1PbMkSNH8lR1bRdqmy+nlJc2nLZJ9b75TXkpW58lP2XrCJns2s/5rR/5EQhlAXo8hvLboW4IIIBAiAg0bdpUli1bJtu3bzfDhBMTE+Wf//ynNG7cWFq1aiUjR440jcEnn3zSzNWo1Y6PjzdzN+rchtrI02Be8+bNZeDAgaK/nAdKOpfijz/+KPv27TNDv+vWrSu33nqr1KhRwye7NurGjRsn2rtw79695lyFChXk8ssvl65du/rktXd0nsfvvvtOdIi0Xl+5cmUZMGBAlmHP+lwa0Hv99dftSwN+f/bZZ7J48WJ59NFHzRD0p556yjSedYi2BlsffPBBc90777xjvvUXdB2Gfc011/j8iq4nJ02aJPPmzZMtW7aYeSX1uTXgq/NrupM2TkeMGCELFiwwAUqdg1Lz9u/fX84++2x3VrYRQAABBBBAAIECFziVNqJOO6NzZWvbUn90LVeunFx44YVy5ZVXmvafu9Ka94svvpCtW7ea9pvOq61tyUApuzac/lg7duxY0Tbp/v37zZzf2ib929/+ZqbL2b17txn1Y/+AvmLFCtOO03buAw88YG6VXdkaDB09erQpW6fU0TnR69WrJzfccEOWdtzLL79s7v/ss8+aZ1qzZo1pc5YuXdrMVeke3h3o+TiGQLgK0OMxXN8c9UYAAQQKUSA5OdncrWzZsuZbG2b6y/O6devk22+/NYFEDUBqw0mTNur++9//mgCZLqRy7bXXmqDj/Pnz5cUXXzS99UxG1x8a1Pz444/NvDra8NRhLhokfPXVV00g0pXVLIajDUgNNvbr188EGzWYqIHFv/76y53V2dZAogbqLr30UunSpYvoPD4ffvihWTTGyWRt6HPlpbelBic1n/3LtgY8e/XqJSVLljQNWt3Wj530V23N79+jUecS0oCkDu3WZ9GGsAYgX3nlFRO8ta/X+2kQU4Otmkfz6tBtddPjOiyehAACCCCAAAIIFKZAftuIGgTUNqL+eKvtMf3RWEeRjB8/Xj7//HOfqmsgT9uB2h7Udqa2D/XH6Pfeey/g6JVAbTitn7aTdOog/ZFWf6xt2LChmcf8tddeM+1Bbd9qm61Tp07m/vrjtO7rooJ2ClS2tunef/99+eOPP0y9tL2rAdRt27aJBhntudLdZeiP9y+99JK5b8+ePU17V3+01gUI165da2flG4EiJUCPxyL1OnkYBBBAIPgC+kvurFmzTMH6C647aSPs5ptvls6dOzuHNSipDTwdbjJ48GDTK9I+2ahRI9Ow0h6St99+u33YfGvj7c477/Rp5GlwUQNz+qv4jTfeaPJp41OHfOsQH+1taCdtHL7xxhum8XfRRRfZh51vbUy6fyFv3769vPnmm+ZXag2O2itYOxfkc+OSSy4xV+gqi9qAvOKKK3ItQRvd+owaSNQekloHbeguWrRI/ve//5m6/d///Z8pR38V1yCjBmTvu+8+p2ztLamByylTphgT5wQbCCCAAAIIIIBAAQrkt42oVdGgoY7geOyxx0wAUI9pm0l/cNXRH9qe0tEcmrQNmJKSYnom9ujRwxzTP2bPni3Dhg1z9nPa+Oqrr8zCftru1LafJm1r6cKJ2h7Vdpi2MbUOOjpG23HVqlXLUzvuhx9+MD/C68raGkC1U8eOHUWDmvqD+PPPP296QdrntJ3crFkzueWWW+xD8uuvv8ovv/xiflxu0qSJc5wNBIqKAIHHovImeQ4EEEDgNAW0R532prOTNvT0l1odlqLntBGlwT530lWu3UFHPafDVfQXag3m6RAVd9Jegdqgmzt3rukFqUNr7KSNTPcvy3pcG4a///67yW8HHnWlQQ0w+i/+okHN8uXLm3tro84/kKg9BN1JG3Y6PEhX9NZh4WdixW4NMGrq3r27T33PPfdc0V/fdQiS9qjUngB20p6T7nTxxRebXqJ2b1P3ObYRQAABBBBAAIHTFQhWG1Gn0tHegPpDtvY6tFOJEiVMUFCHU+u0O9om1Hvq1DI6ukXbj+7UoUMH86Or5skpaXtQf6yuXr26XHDBBT5Ztf20cOHCgD0nfTLmsKM/IGv9tL3qTvp82m7WaXaWLl0qbdq0cZ+W6667zmdff1TWwKP6kBAoigIEHoviW+WZEEAAgVMQ0KEoOm+hf9Lg4FVXXWWCY/7ntKHon+whv/5BR82nwUA9roFJDVC6A4/+gULNHxUVZRqL2stRf1XXeSJ1KLN+dKi0DpvRYd3au1Ibl9oD0x76rNe7kzt4Zx/XhqEGHrWhdyYCj5s2bTJBRW2Ea/DTnTSIqvMeHThwwMxHqcODatasaRrQOmxcA7s67KhKlSrml3P3tWwjgAACCCCAAALBEghWG1HbPZq0Pae9/NxJ23Wa7OCbfmvbToOQgdqb7muz29aFEbXuLVq08PmBV/PrD7Y6b+OpJm1/6hzmOu2Nzuvon/QHbg08apvXP/k/j7b5NOmIGRICRVEg6/9CiuJT8kwIIIAAArkKaA879xBebURVrFjRJziYayFWBh2moim7FZ/tYKM22LSXYm7JXohGG6T2atPaWNWhMZp0ARYtUxtx2kDNT7LnrLQbu/m59nTz6q/02iDWOo8aNSrb4rRRq3MNabD1iSeeMM+tPQB0eLV+NGCqw3v8f03PtkBOIIAAAggggAAC+RAIVhvR/pFVR9PoJ1DSdo8mu21mB+UC5c3tmP7Irel0ysjuHrrojSbt8Rgo2ffU9i4JgeIuQOCxuP8N4PkRQACBEwIauHMPezlVGDvgqJNnB0p2g1Lnz8lL0t6MmuyApTZUdTiK9gC85557zJBkuxxdWdr+pdw+ltO3XUe7cZhT3mCf00Ci9uiMjo428/9kV757SLlu66To+tFekjoUWyc010VybrrpJjOheXblcBwBBBBAAAEEEDgVgWC1Ee0fk3WosQ5FDpTsESp2QE9HvJxqiouLM5fa7b1TLSfQdXbZOjolULIDp3lt7wYqg2MIFBWBk5NGFZUn4jkQQAABBM6ogM6jo8kecu2ujPbu0+Pam1J7KrpToN6KOuREf63W4JzdWF2xYoW5TFcCtHssusvJbjvQEGydS0iT9ig8E0mttGGq82nqkJ9AH7sBrgFXnUzdboDbPR3vuusuU3Uddk5CAAEEEEAAAQRCVcBuI2rPx0BtHj1m/+CqbTOdhkfbaoHaiHl5Rp2ORpMuzueftExdmM9eQNH/fG77VatWNfXT6YACtTHtdvBZZ52VW1GcR6DICxB4LPKvmAdEAAEECldAF2zRhqUuIGMH9uwa6OI1OjRFV3H2nw9H8+ok3e6kDUINyrVt29Y5bAfidG5Gd9JFa+zejrpCtn8aPXq0zyFtEOpK0fpLdDDnd9RgqA6j1vkmc0v2r/26KqJ/o/Wvv/7yaQyvXbtWPv/8c5k4caJPsdpI1+R/vU8mdhBAAAEEEEAAgTMsoKNVdBofDfZpwM6ddETMN998Y+bt1uM6MkTbfzpU2f/HVW1jZtfT0F2m/nCtc2JroFMXrXEnrYO2v9xtVfsH7byUraNWOnXqZNqef/75p7to86O51llH1Oh84iQEirsAQ62L+98Anh8BBBAIsoAOx7n++utl6NChZvXpbt26meCeTiiuk2zXqFFDbrvttix31QbcJ598Yibprl+/vmzZssUEL7XRqHMY2kkboZMmTRINNO7du9dMOq5BuXXr1pmFWjQAp3Mn6urXdtJfzLVRqL94N2vWzEwGrnXRpAvnBFrYxr42v986gbnWb9iwYWbouq7E6B9ktcvUc/PmzRNd3fq1116T888/39RFg6raQNbApK7cqPXT7enTp5tVvnWicg3eamN8/vz5pjj/1cXte/CNAAIIIIAAAgiEgoC29W644QZ5//33TRtRV5bW9pq2z3T+ah3pou0Ze3qdK664wkwr8/XXX8uGDRtMXg1Yarspr203nZ5G21U6LY22u3ShPt3XH7t1uHT37t0dGg0U6o/Reo+ff/7Z5LV/JHYyuTb69u0rOhLn+++/NwFMnbtc22Y6DY7+UH7//febaXVcl7CJQLEUIPBYLF87D40AAggUrIAG93QhlK+++sr00NNgoDbm2rdvbwJ99iIx7lqcc845JuioC6ZoME6TNg7vuOMOn6HQ+svx3XffbRp5S5cuFf3ovJI636MORdaGpAbm3IFHvd/f//53s2r3iBEjTNnaqB00aJC0a9fO7Afrjy5dupig6cKFC0U/up9d4FEbzY888ohob0wNhP7444+mGjq8SBvb+rEb1jqkR1dfVNOVK1eaRrdm1mfXRrw23kkIIIAAAggggEAoC7Rs2VIef/xx+fbbb2Xy5MlmGLUGJOtbPzpre0Z/oLaTbj/66KPy6aefysyZM81hbU/eeeedMm7cOPNDs503u28dbv3MM8/Il19+aX60Tk9PN8FAHaGj7UB7Lkn7+j59+oi2FXWEia6onVPgUeuiZWtPTW27ah11qLiuaH3JJZfQ29FG5bvYC3isuQ3ytASo/g9UuyhXqlTJmWer2OsBgAACCCCQq4D+eq3zGNrz7OR6gZXh4MGDZoiNPa9joGv0/74SEhLMKXuC70D5/I/pHInJycnm/8/8zwVz/8CBA6bxaQ/byUvZeo0OLbJ/6c/uGn127e2pvUE1rx2czC4/xxFAAAEEEEAAgVAT0KlptO2jbUR7Kp3s6qiLDWpMwl7EMLt8OR3XqXi0R6L+wKsjdHJKOnpGg5L2nJM55dVz+iO7Tvmj8ZLcys6tLM4jEC4C+m8xnSZBg/Q5JQKPOelwDgEEEEAAAQQQQAABBBBAAAEEEEAAAQR8BPIaeIzwuYodBBBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CtA4NHXgz0EEEAAAQQQQAABBBBAAAEEEEAAAQQQCIIAgccgIFIEAggggAACCCCAAAIIIIAAAggggAACCPgKEHj09WAPAQQQQAABBBBAAAEEEEAAAQQQQAABBIIgQOAxCIgUgQACCCCAAAIIIIAAAggggAACCCCAAAK+AgQefT3YQwABBBBAAAEEEEAAAQQQQAABBBBAAIEgCBB4DAIiRSCAAAIIIIAAAggggAACCCCAAAIIIICArwCBR18P9hBAAAEEEEAAAQQQQAABBBBAAAEEEEAgCAIEHoOASBEIIIAAAggggAACCCCAAAIIIIAAAggg4CsQ6bvLHgIIIIAAAggggAACCCCAAAIIIBCaAsf2HpCkzfFO5Sqc11wiShZcaCPjeKocmLtEjqzeKMf3HZAy9WpJ9csulpKx5Z06nMmNtKPJcmTlOqcKMU3OkpIVyjn7bCBwpgUK7n+dZ/rJuD8CCCCAAAIIIIAAAggggAACRUXA6xUNMtlJg20RpaLs3Wy/0xKTnHN5vca5IAQ34of/Kovv+5dTsz47Zkp0jarOfjA39v4xWxbd+6wcWbPRp9hus36Sih3O8zl2pnYOLV0tUztd69y+8/hPTWDUOcAGAmdYgMDjGX4B3B4BBBBAAAEEEEAAAQQQQACB3ASSt++W8XW6ONliGtWTnsvGS4noUs4x/430pGQZVe5c53CDe2+U1u8/7+yzkb3A4ZXrZXrvWyUjNS1LpogczLNk5gACxVyAwGMx/wvA4yOAAAIIIIAAAggggAACCISfQOL6LbL6xaFyzpDBZ7zyOhx5w9CvnHpUvvgCiWvbwtkPx43F9z/nE3Qsf05jqX3tZWYYc+la1cLqkfb+OVcSFiw3dY6IKikN7x8UVvWnsuEtQOAxvN8ftUcAAQQQQAABBBBAAAEEECimAmtf/1jq3HCVaFDsTKb05BRZOvglpwrnvvFkWAcevenpsn/GAud5KnVsLV1nDnf2w21jxy+/y/r/fG6qHRlThsBjuL3AMK8vq1qH+Quk+ggggAACCCCAAAIIIIAAAsVTQIcBL7znaRFr/kdS8ASObtwm2ovTTrX6X2pv8o0AAvkUIPCYTzCyI4AAAggggAACCCCAAAIIIBAqAtozb+NH34dKdYpEPY4fPOTzHDFnn+Wzzw4CCORdgKHWebciJwIIIIAAAggggAACCCCAAAIhJ7D8idel5tU9Jbp6ldOq28F5S2WTFcRMWLJakrfvkjJ1akiFc5tKpS7tpN6gviIej0/5G//3reyfvsDqHXjc5/jWb0dLwqKV5ljz5x+Ssg3r+pzPbef4/gTZ+ME3cmDOEtFVm8ueVVti27WUsx+7O7dLfc4f271PNrz3teydOkeObo6XUlUqmufRFanPuvNv4ilRwif/5s+Gy94ps+XYvgM+x9e98YnEfz/OHKtlzfOo1v5p5+jJsn/WIuu5V4guTKP3irGeu2a/3lLnuj5Z7PR6fba1r33sFNXs2fslpknWIOeyx16VlB17TL6481tKo4duda7JbuPYnv2y9JHM4e8HT8zvqHnTU47JvJsy5wWt0KqpNHn0ruyK4DgCQREg8BgURgpBAAEEEEAAAQQQQAABBBBAoPAEqnTvKPusRUN0PsLUhMOy5OF/S/vv/3tKFdBg1MI7/ylbvxnlc70GuzT4t+njH2Tzpz9Ku89fk7IN6jh59s9cmOUaPZmwcIX56HbDB2/OV+DxyKoNMuPKu+Tohq16uUlJW3eILpCy7ZvRUqVre/twjt9bv/pFFt79lAm02RmTt+009doy7GfR8xd8+7aUqVvTPm2e1d9AT+q97VSueSOfwGPqoSMy/7bHZcfIiXYW823fK374r7LuzU+l05iPsgSGU3bu9fFrcO8NAQOPO8dOEXXRpO8qL4HHtMQkn7LNxdYf3rR053j1AwkEHm0YvgtMgKHWBUZLweEucODgQYmP3xHws3jJMpk5c064PyL1RwABBBBAAAEEEEAAgTAViG3dXBo9fKtT+/gfxsmuX/909vO8Yc0POfOKu5xglH1dyfIx9qb53vfXPJnc9mqn550eLFEqSkqUKS0lSkf75I0oGZl53Drnich72EF76f3RcYBP0FELtnsmpuzaK9u+H+tzr0A72mtz3s3/8Ak6RpYr65Sj1+gQ9amd/ya6MI6ddMVn8zzRpexD5jvCfk7refTZ7KT1ndzmKp+gYwnr2vJWcNKdDs5fZnoZejMy3IcLdNsT4Tn5DiJ9e3aaZ9RnsZ6LhEBBC5z8X0xB34nyEQhDgedeeCXbWsfGxUmnTnn7tS3bQjhx2gIJR1IkNT1DKlco4z/y47TLpoC8C+xNSDKZq8SWyftF5AyqgE4pv896DyVLREhsOd/Gf1BvRGE5CiSlpMpR61OhbCmJKunbyM/xQk4GVUD/t6D/m+C/SUFlpTAEEAhBgebPPyzbf/pNkrZsN7Vb9H//kl4rfjUBp7xWd/27X8qeyTOd7FV7dpbW7z8vMY3riw553vDeV7LyuXfMAjbas3LR/c9JxxHvm/xtPn5J9KO9/kbHtnbKaPHyo9J48B3Ofl43dJVuLctONa/pJS1eGiwxjeqbIczr/jNMtlnDuHNKujDM0keGOFkqtDxb2n35hsSe10wyjh2XzZ//JEseetEsHpMcv0s2WM/f5MQQ7vPe/Zfo58DcJfJH+/5OGR1+Gio1ruju7Nsba175n+j97KSreZ/1/26QyLKlTfB06aOvOEFJNd5rfape0sXOXqDfZerXlr5Hl5l7LPn7EJ9Vra8+srRA703hCLgFCDy6NdhGwE/ggfsDzyEyZtzvkmD1iCSFhsDhxGMm8BgatSmetUhN1V9vWU3xTL99/d9CpQqlz3Q1iv399T1o4JF0ZgWOJqdagcczWwfujgACCBS0gAa4Wr/3nMyweixqSrLmMdQgYcvXHs/TrbUH3ppXPnTyanCuy6+fiedED7moSrHS7NkHxJvhlVXPW8FHK+mQYg2iVe3RybkuGBv2vI52WZU6txUN+Nk9JuPOP1fO//J1E5jcNe4PO1uW73VvfSppR5Od4+2+eN0EHfWA9vBrYAUGj26Kt+ZW/MjkWfPqR1bP0dtEezvmN2k5Ov+kJg30NX7kdmcuR53Xstkz9zuBR81zcP7yQgs86v1ICISCQN77PIdCbakDAoUoUNHq0djq3BYBP+XLl5eoUvyjshBfB7dCAAEEEEAAAQQQQACBAALV+3ST2taCJ3Za9/ZncmjJKns3x2+dIzJlZ+aiJZqx8eA7naCj+8LG1pBuHapsp71/zLY3g/atw5HdAcNGD97iBB3tm+iQ6+qXd7V3s3zrfJc6p6KdNFipQ9L9U90brnQOHbfmOdSA7amkjiM/kEs3TjWfi6Z87QQd7bL03u4Ff46s2Wif4huBYiNAj8di86p5UAQQQAABBBBAAAEEEEAAgaIo0Oq/z8juidNNb0BdPGTh3U9L11nDc31Ue+VpO2PVHh3tTZ/vkrHlJa5tC7M6tJ5wr5Lsk/E0do5uOjlkWYvJ6yIy7lsmbd0pOu+inTTwt/u3v+xd5zvjeKqzrRs6XDrQatI+mXLZ0R6bOkRbh2/rJ+1IorkiPelk70sNjJIQKG4CBB6L2xvneRFAAAEEEEAAAQQQQAABBIqUQHSNqtLilUdl0b3PmufSANjG976W+ndcm+NzJm/f5ZzX4dWlqlV29v03ytSr5Rw6tHS1sx2sDR22bCdTlyoV7d08f6fs2O2TVxeZ0U9uyT/omVt+93ldcXvNy/+TLV+M8Fmoxp2HbQSKswCBx+L89nl2BBBAAAEEEEAAAQQQQACBIiHQ4J6BsvXLkbJ/1iLzPMufelOqXXpRjs+mczfaKTLGWvU5hxWo7XkfNb/2qgx28qalOUWWLGetqO3xOPt53Ug7cjSvWX3ynep12rvyr66Zc0baBeqcj2WtxXCiKlawHsEjO8dM9hlCbufjG4HiIkDgsbi8aZ4TAQQQQAABBBBAAAEEEECg6ApYQa42Hw2RyW2ukozUNGuo71FZ/MDzOT5vuSb1nfO6YrUOCy5RJvBCdcnbT/YmdM9b6BRwmhtlrcVZ7HT84CFJTzkmJaLzN69+2QZ17CLMd9N/3iu6SE1uqVzThrllCXh+wR1PmoVq9GRkTBnpMHxolmDvr2d1tQKPJ3tzBiyIgwgUYQECj0X45fJoCCCAAAIIIIAAAggggAACxUegfIsm0vgfd5qhv/rUuydMDi6wtAAAQABJREFUy/Hh/ec1PLxqg5nLMctFXq8csc7ZyV7J2d4PxreuCu1Oies2S4WWZ7sP5bqtK0nrytUZx447eXNajMbJdAobuiL43qknF9lp8tjdWYKOuRXr7kWqed2L6+R2LecRCBcBVrUOlzdFPRFAAAEEEEAAAQQQQAABBBDIRaDZM/eLBuDykjSwFxFV0smq80IGSrvGT5WkLdudUzWu7ulsB9rIcA2bDnQ+0LHy5zT2GV69+ZMfA2WzFm7ZGfC4HtRVr005J3JsHvazaE/OQEl7hR6YvTjQqTwd0wVp0hKTnLwly1vDw/1SenKKpB89mcfvtJSpW9PnkDuQaZ/QIGra4cyFauxjp/PtTc84ncu5FoF8CxB4zDcZFyCAAAIIIIAAAggggAACCCAQmgIlSkdLmw9eyFPlSlWtJI0evs3Ju+WrkWaRFOeAtZG4dpMsfvBkeVFxFaRWv97uLKJBN/ew6IPzlvmcz8uODpOudU0vJ6suCrPjl9+dfd3YMXKirH31I59j/jvnvPCwcyhlxx6Zc/1DPitd60ldgXr6JTfL1Auvk23fjnby52ejdK1q4u6xuOnjH8S9Wrb2XpxxxV1ybO8Bp1j/uTF1wR73vJo7R0/Okn/OdQ+Ke5i7U1g+NqKt92wnDYYeXr7W3uUbgQIXYKh1gRNzAwQQQAABBBBAAAEEEEAAAQQKT6DqJV2k7k1Xy9avR+V602ZP32cWpUnZtdcsGjP/1sfMStCxbc4xvRz3Tp1j5ou0C2r1zrNSskI5ezfz25pfUgOHh1euN/vbf/5Nppx/jZSpX0saP3KHVOrY2jd/NntNrbpsHzHBnNU5Hmf3v0/izm8pOiT8wJwlJgiazaXO4ep9ukntv10u8T+ON8d0uPnEZr2lSrcOEl2zqhxatFIOLljurEC94tn/SI2rL5HIsoHntnQK9tvQAG+Vbh1lz+/TzZnDK9bJ2GrtpcYV3Uxgc9+0eT49IjXTMSvg6U7a27Rmv16y/affzGEtY3ztzqLPoD01D85f5mPvvjY/22Ub1fPJPvXC66Vq944SbQVPz7PeJwmBghQg8FiQupRdpAUmluku+8bsK9LPGA4Pl2qtqHcsNV1iSgd/Zb1weP5QqWPysTRTldKljoVKlQqsHqNXJ8mYG6rJ+bXyN9l5gVWIghFAAAEEEEAAgQAC5771lOwa/6ccP+Ab7PLPGlmurHT+9TOZO/BhObI6cx7H/TMXin7cSXvmnTPkERPQdB+3txvcN0gW3/cve9cEzTRwVu+Wfs6x3DZiWzeX84Y+J0seelG86emi8yhqwFE/dqras7PsmTTD3g343fp/L5rjdvBRDTQY6p/izj9XOo/5KN9BR7ucdp++LJNaX2kCjXpMg4XuYK8GaPWTtHWHueToxq32pc53iyGDZde4qU4gVHtNas9OO0VVipVIa5XvpM3x9qF8f9e4srsZ1m3XQ+upAd64ti3yXRYXIJBfAQKP+RUjPwIuAQ1AkEJFIDPwFSq1Kb71OF58H50nRwABBBBAAAEEQkigVJWK0vL1x0VXXs4txZ7XTHos+EWWPfG6Gd6cvM01j6LVm7H6pReJLp5SpWv7bItqcPf1pofkujc/NUFDO6PHuj4/qeF9N0mMNUfl0sEviS52I9bCNpp0aHPL15+Qiu3Pk98adsuxSB0O3v6Hd6T2dX1k1QtDrYVx1jvDoLWXoQY4q3TtIM2evT/bVbxzvMGJk6Xr1JCei8fK0n+8nNnD8kRdda7JKlaPwrYfD5EtX46UlVavSk1HN2wVXTQnpnF9s69/aG/OHgtGyfzbHvMJsGo9tZdm209elqWPDDmtwKMOg+8w4n2ZN2iwzyJBks9341SaDQTyIeDxWikv+dOtXxvi4+OlUqVKEhOTddLUvJRBHgSKisA7730qXx5uLvGRtYrKI/EcCCCQD4FQ7PGo/2e+Mf6gVKpQWmLLRefjacgaTIGklFTZuS9R6lQrL1ElSwSzaMrKh8C+hCRJTE6V+jUq5OMqsiKAAAII2ALaQ1CH/WpPO13BOsvQajtjgO9UayEU03PSapzENK4nURVjA+TK2yHtmXfImo8wxhoqHF29St4uCpBL51Y8smajtWp0ksS2amZWvg6Q7bQO6dyJR9ZssgKcx81q3DoUO78p9dARM1w9MqaMlGvaUCJKBrevmPYgTdq83fTALF27upS1VhJ3z1OZ3/qSv3gLJCRY/504fFjq1s15Mavg/i0u3uY8PQIIIIAAAggggAACCCCAAAJhL6DBwsoXnn9Kz6ELzVS8oNUpXet/UcnY8lK5Szv/w/ne1+Cae7XrfBeQhws00Kg9R08naYA3r/Nhnsp9dLi8zsWpHxIChSXAqtaFJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCBB6L0cvmURFAAAEEEEAAAQQQQAABBBBAAAEEECgsAQKPhSXNfRBAAAEEEEAAAQQQQAABBBBAAAEEEChGAgQei9HL5lERQAABBBBAAAEEEEAAAQQQQAABBBAoLAECj4UlzX0QQAABBBBAAAEEEEAAAQQQQAABBBAoRgIEHovRy+ZREUAAAQQQQAABBBBAAAEEEEAAAQQQKCwBAo+FJc19EEAAAQQQQAABBBBAAAEEEEAAAQQQKEYCkcXoWXlUBBBAAAEEiqyAx3qyutUrWH96i+wz8mAIIIAAAggggAACCCAQXgIEHsPrfZnajhs/UVatWiMtWjSTS3v3zPIE+/YfkGHDvpGLLuosF5zfJsv5gjww7ItvJTo6Wq6/rl++bpOWli4//TxKli1bIc2bN5Ubb7g2X9fnJfMfU6fJggWL5R+DH8hLdvIggAACQRHYu3evzJ8/X+rXry9nn322RETkPtjgyJEjsmXLloD3b9y4sZQqVco5l5aWJuvWrZOVK1dK6dKlzT0aNmzonGcDAQQQQAABBBBAAAEEEDhTAgQez5T8adx3x45dsnrNOlm7boOcc04zqVO7lk9px48dM+f1XH6TBjX//GuGDHnxaSlZsmR+L5cdO3ZKmTJl8n3dmrXrZNLkqdKwwVnS9OzG+b4+Lxfs2bPXuNh5V69eK59ZAdo7brvJ+od6wdzTvhffCCBQ/ASOHz8u999/v8yZM0e83sxeiNWqVZPPPvtM6tSpkyPI5MmT5dlnnw2YZ8SIEdKoUSNzbt++fXLXXXfJhg0bxOPxOPe57rrr5Mknn8xTkDPgTTiIAAIIIIAAAggggAACCARBgMBjEBDPVBElSpSQr776Xp584hHzD85g1CM5OVkOHDhY6AP14uN3mOrfftuNUq1a1WA8Sq5lpKammWc9npqaa14yIIAAAvkVeP/992Xu3Lnyr3/9S3r16iWbNm2SRx55xAQKf/75Zylbtmy2RW7dulWqV68uTz/9dJY8NWrUcI499NBDEh8fL2+99ZZ06tRJ1q5dKx9++KH88MMP0rRpU+nfv7+Tlw0EEEAAAQQQQAABBBBAoLAFch/vVdg14n55FriiT2/ZuGmLTP1zep6uSUpKlnnzF8mo0eNNr8a9e/f5XLd+/UbZu3e/ObZixSrZvXuPz3n/HR0evc7qdfnXtJmyZes2p6eNfz7d3xa/3eSbMWO2dQ/f++o57Y2oKX77Dtm1a7fZtv9ISDgkM2fNNc+5ffvOLPfR67W+/mnz5q2mV6j/cd3Xe2zanDmMUfNt2LgpUDaOIYAAAqcksH37dhk2bJgMGDBA+vXrJzExMdKyZUsZPHiw1TN8h0yZMiXHcjXw2KRJE2vKjIuyfOyApQ7HXr58udx6663Ss2dP09u8+TktTU9HLXzs2LE53oOTCCCAAAIIIIAAAggggEBBC9DjsaCFC7D8S3v3kLnzFsiIkWOlTetWUqFC+WzvpgHK/334mRw8mCCVK1cyPf10WN6A/ldJzx5dzXU/DB8pGoTT9MH/PpPLL+slfa++3Oz7/6H/4P3vux+a/FElS0qqNcdYkyaN5Jg1tNA91FrnHvvu+59NoNMeup1q9TDsdUk3uXZAX9NT84cfRsiatevNLT78aJh0aN9ObreGP2dkZMh7H3wiS5YsN/OZRUVFid43NraC/OOR+63eQNXMNRMmTpFFi5bIe+++4VPNkb+MlQMHD8qLzz/lc1x3plsBUL1O0+gxv5o5Mx979CGzzx8IIIDA6QqsWbPG/DesR48ePkVpL0RN06ZNkyuvvNLnnHtHA48XXHCBOaTDtPW/1/5p2bJl5vi5556bmc/68/DRY1KrVi0pV66cJCQk+F/CPgIIIIAAAggggAACCCBQqAIEHguVO7g306HWNw+6Xl559T/y/Q8/yz133xbwBhro06BjtLUYweuvvmACdzq8+LPPvpYffhwpZ9WvJw0bniVPPTnYLPDy24TJMvTd10UDitmlz7/4zuq1s0vu/X+3S+vzzpWjR5Pkk0+/FO2RGBcb61w2/tffTdCx79V9pGvXLlLaWnhm8pQ/5cfhv0j58uWtxXF6mMVefpswybr3aHnrjSFWz6DM4Ye6EIwGHTVA2fXiLlbwMUp0XsY33hoqv0+aKoNuus65T343BvS/Ws5u0tgKnv5PHnzgHmnZonl+iyA/AgggkK2AvTBMgwYNfPLUrl3bzLu4e7dvz26fTNbOtm3bpGbNmjJw4EBZv3696THZoUMHM1S7SpUqJrsOrV68eLHPpRXLl5aJEyeYH2m6devmc44dBBBAAAEEEEAAAQQQQKCwBRhqXdjiQb6fLsaiQTkdQr08wHBjvZ0OU9Z5Gy+9tKcJOuoxDSrecstA3TSLupiNPP6xc+duWbp0udVT8mJp2+Y884/ocuViTC/FyMiTsezklBTRIGaDBvVFh4XHWPOZabC01yXdpV69Olavw1k53rFM2TJy2aWXmN6RGnTU1LRpE6lcqaLst1buJiGAAAKhKrB582ZTtQoVKvhUUf8bqT0SD1q9sbNLBw4csH7MOWp6RTZr1kweeOAB67+jDWT8+PGii8bo+UBJ+0ROmPCbvPDCC1YP+ArmukD5OIYAAggggAACCCCAAAIIFJbAyShRYd2R+wRdoF+/K61eL0vl629+lBee/2eW8rdt226OnW0NhXYn7X1YtWoV2ek3p6I7T6Bte25E/1Wzdah3ddfCMDutHpG6qmtcXKxMm+4bZNR7b9myzQxFjIgIHP8+p3lT0Y/Ox6greGuwccOGTbLP+raHWQeqH8cQQACBMy2QYv3woik9PT1LVXTYdCmrB3p2SYdW60rV2qOxbdu2JtvNN99sLSb2lbz++uuii9b4LzqjQ7PfeOMNmTp1qplL8uWXX7YW6sqcjiK7+3AcAQQQQAABBBBAAAEEEChoAQKPBS1cCOVrEG/gwGvlfWs+xDFjfpOOHdr53PWANa+jzrtYyeop6J/KlCltDclL9D+c477OE6mprFWmfypr9VK0036rl6UmXbQm/kTw0z6n39WsoKf+49w9J6T7vK50rUPIV69ZJxWsYdnVa1STRtaQ8NwWvbHL0KAnCQEEEDgTApUqVTK31UVmdJEYdzp06JC4V6Z2n9NtvVZ7Ofqnq666Sv773/+Kzu3oTt999528+eabppfjc889J9dcc03AOSHd17CNAAIIIIAAAggggAACCBSGAIHHwlAuhHu0aX2umWtx4u9TpE6dWj53jLV6IiYlJZkeg+7go65Krb0hGzXynYPM5+IAO7q4i6aNmzZL7do1fXLoCtRVqlQ2x+JO5Otmze14RZ9LffLlZeezYV+LBh8f/ceD1nyMJ3trzpkzP8vlVgehLGm3tVK2OxCaJQMHEEAAgQIS0OHUmnSuRnfgURd8OXz4sDRu3DjbO+/Zs0cSExPN8Gp3Jl0Zu3Tp0j69JT/44APRT9++feWxxx6z/ptXVlKOpVnz6WY/R6+7TLYRQAABBBBAAAEEEEAAgYIUCDzGtSDvSNkFJnDDDdeauRuHW6tTu1PdurXN7iprYRZ32rJlq+iq0+6gnvt8dtv16tYxp7QnojvFb98hGuyzU82a1UXnM1u6dIV9yPl+01og5vU333X2/Td0qKEGHZs1O9unfjrMWj/upIFV7d2YmHjUObxj5y7rH/dHnH02EEAAgcIU0NWsdUj1lClTfG47YcIE0f++de3a1ee4e+enn34ygURdVMadZs2aJdpbskWLFubw3r175cMPP5TOnTubeR01MKn3JOjoVmMbAQQQQAABBBBAAAEEzqQAPR7PpH6Q7609DHW+x2++He5TcudO7UVXl/5l1DipWDHOWsW6rjWv4x75/ItvrWHOpaVH94uc/DpPo6bZs+ZZ84Q1N/MzOidPbGgvx7Ztz5P51oI2urhN2zatrIUSEkx59iIwmlWHUF/au6eMGz9Bfv1tknTqeIEVIEyVCRMniwZB77x9kH/Rzr7+41nncVy3dr2sWLla6tSuJeut+R2H/zRSoqKi5KjVgzMpKdnUv771PJq+/3GEXNKzq+zbu19GjhprVtB2CgywYT/r4sXLzDD0mjWqB8jFIQQQQCD/AtWrV5c+ffpY01+MkbPOOks0ELl8+XJ56623zPb555/vFPr444/LggUL5OeffzbDpXv37i2ff/65PProo6LndGEZDULqvI3639XbbrvNXLtw4UIzT672rvzkk0+c8uyNuLg46d+/v73LNwIIIIAAAggggAACCCBQ6AIEHgudvGBvqCtcz549XzZs3OTcqKS1gvXfH/o/+fCjz+Wtt99zjusci48/+rD5h6x98LxWLa2VVGfJl19/L30u7yXX9L3CPuXzfcuggaaX4Xff/yT60UDh1VddbhaMSU1NdfJeeUVvq1dlqowYOUZ+HjHaHNf5GgfddJ106HDyH97OBa6NW28eKN9+95O8/Z/3zVFdFfuGgQPk8JEj8uPwX+Q9a07LRwc/IG1atzJBTV29e/bseaIB2C5dOsq+fftls9WrM7ukQ9J18Zq/ps0U7a355ON/zy4rxxFAAIF8Czz//PNmmot3331X3nnnHdGFtNq1aydDhgzxmYNRV7jW4dXaE1JTw4YNzZyNL730ktx9993OfTWAqQHJKlWqmGMaeNT0448/mm//P7QcAo/+KuwjgAACCCCAAAIIIIBAYQp4rH/oBJgdL2sVdGXO+Ph4M+m9DucihZ+AvupdVk9HDchVq1bFzMWoAcNA6YC1MEz58uXMUOlA5+1jOrRa89azhnNnt0iM5tVh0Fu3xUsJ6x/e9evXs+Yoi7KLyPV7587d1j/IM6zFGKo7/1jXuSQ9ER6z6IxdgPaA1LrUqlXDyWefy+lbF9eJLBmZaw9JdxnvvPepfHm4ucRH+s6n6c7DNgIIFF2BMTdUk/NrZb8ytfvJdV7HVatWWVNHNJPY2Fj3qRy3dSoM7emo19erV8/qBX7yv4E5XsjJMyqQlJIqO/clSp1q5a3pT0qc0boU55vvS0iSxORUqV8jc17q4mzBsyOAAAIIIIAAAgUhYM9fX7du5ijU7O5Bj8fsZIrgcQ0y1rBWhtZPbkmHZOclaa9J/eSWYmLKSnNrvsZTSYHqay9w4y5Ph43rJ7+pXDkC6fk1Iz8CCORdQIONHTt2zPsFJ3LqHLlNmzbN13UHD6dYP6JESnQU//eeLzgyI4AAAggggAACCCCAQIEIsLhMgbBSKAIIIIAAAoUroMMXDhxONqtaF+6duRsCCCCAAAIIIIAAAgggEFiAwGNgF44igAACCCCAAAIIIIAAAggggAACCCCAwGkIEHg8DTwuRQABBBBAAAEEEEAAAQQQQAABBBBAAIHAAgQeA7twFAEEEEAAAQQQQAABBBBAAAEEEEAAAQROQ4DA42ngcSkCCCCAAAIIIIAAAggggAACCCCAAAIIBBYg8BjYhaMIIIAAAggggAACCCCAAAIIIIAAAgggcBoCBB5PA49LEUAAAQQQQAABBBBAAAEEEEAAAQQQQCCwAIHHwC4cRQABBBBAAAEEEEAAAQQQQAABBBBAAIHTECDweBp4XIoAAggggAACCCCAAAIIIIAAAsVTYMyYMTJq1Kji+fA8NQJ5FCDwmEcosiGAAAIIIIAAAggggAACCCCAwOkJ7NixQ95++20ZPnz46RUUAldPnTpVJk+eHAI1oQoIhK4AgcfQfTfUDAEEEEAAAQQQQAABBBBAAIEiJTB9+nRZvXq1TJkyRQ4dOhSUZ1u7dq18/fXXsnLlyqCURyEIIBA8AQKPwbOkJAQQQAABBBBAAAEEEEAAAQQQyEYgLS1N5syZI1FRUZKRkSGzZ8/OJmf+DmsvymnTpkl8fHz+LiQ3AggUuACBxwIn5gYIIIAAAggggAACCCCAAAIIILB06VJJTEyUyy+/XMqVKyczZszIE4rX6zW9I/W7MJP2yDx+/Hhh3pJ7IVDkBCKL3BPxQAgggAACCCCAAAIIIIAAAgggEHICGmj0eDzSsWNHE0j8448/ZMOGDdKwYcOAdV23bp0Zkq1Ds5OSkqRUqVLSuXNnufrqqyU6OtoELn/44QdJT0831+tCL2PHjpXLLrvMfA4cOCDPPfecNG7cWB544AGfe8ydO9cMz7bz2iePHj0qWqYO2z5y5Ig5XLt2bbn22muladOmdja+EUAgjwL0eMwjFNkQQAABBBBAAAEEEEAAAQQQQODUBBISEkwwT4N3sbGxJvioJWXX63Hbtm3y7rvvmvkgNdh41VVXSVxcnAlEvvfee2aotgYEe/XqJc2aNTOV0gCj7tuBTO0heezYsYC9FjVYqefsoKUWcPjwYRkyZIjMnz9fGjVqJH/729+kTZs2snv3blMXhnIbZv5AIF8C9HjMFxeZEQgsEB3pCXyCowUukDnawmt+OS3wm3GDbAWKy3tISSvc4T3ZgnMCAQQQQAABBBAIMwGdz1HndezQoYOpeb169aR69eomyHfdddeZ3oz2I2kwcOjQoSb/U089JdWqVTOn+vTpI++8846sWLFCFixYIOeff75oObq69LJly6R58+Ym8GiXk99vreP+/fuld+/e0q9fP3N5jx49ZNy4cTJ69GjRhXGuv/76/BZLfgSKtQCBx2L9+nn40xHolTRFXnrxqdMpgmuDIJBwJEX2H0qWBrXirOBjEAqkiFMS2LE30brOKzWrlDul67kIAQQQQAABBBBAoGgLzJw50ywq07p1a+dBNQj5yy+/mCBip06dnOO6WIz2kLzwwgudoKN9UodG69yLOow62Omiiy4yPTHLlCnjU7T2etTA486dO32Os4MAArkLEHjM3YgcCCCAAAIIIIAAAggggAACCCBwigLr1683w5Xbt2/v07PxggsuEJ2XUYOS7sDj5s2bzZ3q1q2b5Y46nPqZZ57JcjwYB3TeSP3oMG+d4/HgwYNmbkl7gRn3sOxg3I8yECgOAgQei8Nb5hkRQAABBBBAAAEEEEAAAQQQOEMCGljUpHMuTpo0yacWurq1LiKzZ88eqVq1qjmncypqKl++vPkurD80sPjxxx/LokWLRHs91qxZU8qWLSupqamFVQXug0CREyDwWOReKQ+EAAIIIIAAAggggAACCCCAQGgI6AIuuliLJl1JWj+BkgYn+/bta07p4jOaEhN1Op/CS2PGjDFBR53jUReziYzMDJloz8cnnnii8CrCnRAoQgIEHovQy+RREEAAAQQQQAABBBBAAAEEEAglAV0ERoOPOn9iz549s1RNexO+8sorMmvWLBPsi4iIcHo+bt++PUt+nftx3rx5UqdOHdEVsnNKOmxak16Tl6TDqzVdfvnlTtBR97WnJgkBBE5NIOLULuMqBBBAAAEEEEAAAQQQQAABBBBAIGeBGTNmmAwaeNTVqf0/tWvXllatWpngoB34a9asmcTFxZm5H7W3oTuNHDlSfvrpJ0lOTnYOx8TEmO2kpCTnmG7oMGkNPuowbl2t2k4a7Jw2bZq963x7TqxWuXr1aueYBh1HjBhh9tPS0pzjbCCAQN4E6PGYNydyIYAAAggggAACCCCAAAIIIIBAPgQ04KcLy+hcidpDMbukq1vrcGwNUrZo0UJKliwpAwYMkE8++UReeukl6datm5lzUedeXLNmjTRp0kRatmzpFKeBSu0pqddHRUVJgwYNnN6QHTt2lD/++ENee+01s4CNBh1XrFgRcIVqXfxGF7b54v+3dx9wUlXn/8cfOktbeu9IERAEAbEjKGJsxPbXGBJ7iy3+jOhPTTRYkl8CEUtiwYINpSggKtJEBAWkI1KlSO8LuyywoPzv95A7zszODLOVmd3Peb2Wmbnl3HPfZ3eZfeac5wwdaosWLXKBy++++862bNnirqX7oSCAQM4EGPGYMy+ORgABBBBAAAEEEEAAAQQQQACBOAT8RWUUWIxV2rVrZ1pkRsE+P69jly5d7L777nMBx7Fjx9qwYcNMU6979Ohh9957b8hUaI1svOiii0yLw2iV7MWLFwcud9lll5nq2rNnj3366aducZvU1FS79tprA8f4T3r27Gk6XiMbFcScOHGiC2Q++uijrh379u2z9PR0/3AeEUAgDoES3rDhuJIV6Ad4w4YNVqNGDfOHMcdRP4cgUCQFnnvxNfep19MDHimS95dMN5WWfsB27tlvzRtUs//OjEim5heZtm7arsTfR6x+rcpF5p6S7Ub0n/nqDbutRmqKVa18NJ9Rst1DUWhv5oFDtnlHhjWqU8XKlilVFG4pKe9hR1qmZew/ZE3rpSZl+2k0AggggECowIEDB1xAsmbNmqE7wl4pYLhr1y4XtyhVKvT/YeWZ1LRtxTQ0ojJWUfxjx44dLhiq1a0pCCCQXUC5U/fu3WuNGzfOvjNoC1OtgzB4igACCCCAAAIIIIAAAggggAACiSWgPI3+QjGxWqZVqGvXrh3xkHLlylndunUj7gvfqKClclFSEEAg7wIEHvNuSA1FVGD37ugrnykvyM/ep2AUBBBAAAEEEEAAAQQQQAABBBBAAIHIAgQeI7uwFQFv0ugRe7D/X6JKVK1aNeo+diCAAAIIIIAAAggggAACCCCAAALFXYDAY3H/DuD+YwrceMNvI+6fOHmapXu5DCgIIIAAAggggAACCCCAAAIIIIAAApEFCDxGdmErAla9WjU7/bRuESXmzFtsSk5MQQABBBBAAAEEEEAAAQQQQAABBBCILFAy8ma2IoAAAggggAACCCCAAAIIIIAAAggggAACuRcg8Jh7O85EAAEEEEAAAQQQQAABBBBAAAEEEEAAgSgCBB6jwLAZAQQQQAABBBBAAAEEEEAAAQQQQAABBHIvQOAx93aciQACCCCAAAIIIIAAAggggAACCCCAAAJRBAg8RoFhMwIIIIAAAggggAACCCCAAAIIIIAAAgjkXoDAY+7tOBMBBBBAAAEEEEAAAQQQQAABBBBAAAEEoggQeIwCw2YEEEAAAQQQQAABBBBAAAEEEEAAAQQQyL1A6dyfypkIFG+BnaWq2Xfbsoo3QgLcffq+Q7Yn4yfLLJNlJUokQIOKaRO27z7s3fkR23UksX4mGlQubdVS+IytmH5bctsIIIAAAggggAACCCBwnAUIPB7nDuDyySswt1wnO2/oluS9gSLX8swid0fJeUPpCdXspXc1TKj20BgEEEAAAQQQQAABBBBAoDgJMAykOPU294oAAggggAACCCCAAAIIIIAAAggggEAhCRB4LCRoLoMAAggggAACCCCAAAIIIIAAAggggEBxEiDwWJx6m3tFAAEEEEAAAQQQQAABBBBAAAEEEECgkAQIPBYSNJdBAAEEEEAAAQQQQAABBBBAAAEEEECgOAkQeCxOvc29IoAAAggggAACCCCAAAIIIIAAAgggUEgCrGpdSNBcBgEEEEAAAQQQQAABBBBAAAEE8i5w+PBhW7t2rW3ZssUyMjKsRo0a1q5dO6tQoULeK8+HGg4ePGibN28O1FSnTh1LSUkJvOYJAsVJgMBjcept7hUBBBBAAAEEEEAAAQQQQACBYwgcOXLEsrKyIh5VpkwZK1ny+E2eXL58ub377ru2devWkPb179/fmjdvHrLteL3YuHGj/f3vfw9c/u6777b27dsHXvMEgeIkQOCxOPU294oAAggggAACCCCAAAIIIIDAMQTS0tLsoYceiniUgo6pqalWs2ZN69y5s3Xv3r3QRhpqFOHgwYPtp59+ytY2BUQpCCCQeAIEHhOvT2gRAggggAACCCCAAAIIIIAAAgkp8PPPP9vu3bvd18qVK+3DDz+0Sy+91Hr37l3g7R02bFhI0LF+/fp2yimnuGnMVatWLfDr5+cFVqxYYT/++KOrslSpUnbuuefmZ/XUhUDCCBB4TJiuoCEIIIAAAggggAACCCCAAAIIJJ5AuXLlrHz58q5h+/fvD5mGfejQIRs1apStWbPGbrrpJitdumDCDAp4/vDDDwEcTavW9OpkLQsWLLDJkye75suXwGOy9iTtPpZAwfxGONZV2Y8AAggggAACCCCAAAIIIIAAAkkhcPbZZ9uVV17p2qr8j9u2bXNBQI12TE9Pd9vnzZvnpl9fccUVBXJPO3bsMC0q4xdN86YggEDiCxy/jLCJb0MLEUAAAQQQQAABBBBAAAEEEEAgSKBEiRKmVZpPP/10e+yxx6xZs2aBvRMnTjRNvy6IkpmZGVKt2kBBAIHEF2DEY+L3ES1EAAEEEEAAAQQQQAABBBBAIOEEtMhMv379bMCAAaaRkPpS8LFly5ZR27p3716bOnWqKcehRjFWrlzZGjZs6AKYZ555ZrYVs2fMmGFayTojIyOkTl1nzpw5bpvyPHbs2DFkv14sXLjQVq9ebevXr7dNmza5a9WqVcs6depkXbp0MQVRw8uGDRtswoQJgc0XXXSRC7QGNvz3iaaX79mzx71q0qSJ9erVK/yQbK81OnTEiBFu+7p16wL7NV399ddfd69lURj5MgMX5wkCBSxA4LGAgakeAQQQQAABBBBAAAEEEEAAgaIq0KBBA7e69dy5c90tLlmyxPbt22cVK1bMdsszZ860d955xxRo84sWqtEiK19//bVp/80332zVq1f3d7vckbNmzQq89p8ocOmXevXqhQQelYdy6NChNn/+fP8Q9+hfS21V4PKuu+6yKlWqhByjYGLw9c4555yIgcfFixebVtlW0f3EE3g8ePBgSN3+hZW/0r+m7Ag8+jI8FgUBploXhV7kHhBAAAEEEEAAAQQQQAABBBA4TgLt27cPXFl5GDVCMbx89dVX9sYbb4QEHbVgTcmSv4QltHjM//3f/4Uco8VqypYta2XKlAmp0t+ufVoV2i8aVfjkk0+GBB11roKTwUUjDl977TU3SjN4e0E+1whLtVdfwfeta/rbC2pxnoK8L+pGIJYAIx5j6bAPAQQQQAABBBBAAAEEEEAAAQRiCgSPUNSBGlkYXDSl2p9irO0aJXnDDTdYo0aN3IIxGu34wQcfuOc6d8qUKXbBBRe4Kq655hrT19q1a+2ZZ54JVHvbbbdZhw4dAq/9J+PHj3dTuP3XWhRHi+No5ejt27e7Fbj9kZDLli0zfZ144on+4QX6WKNGDXv++efdNYYPHx6yqvVzzz1XoNemcgSOlwCBx+Mln4TXnfLFNJs3b2FIy/XJUv36da1J40ZejoxOlttPZw4f/slGjhpjixcvsbZt29h1v7kq5Dr58eKLqV/Z3LkL7IH/uTs/qqMOBBAoYgL6hF1TdvSJvd4E56ZoNcfvv//efvvb32Y7XW+4FyxYYDt37vR+b9a3bt26uTfA2Q5kAwIIIIAAAgggkGQCNWvWDGlxWlpayGtNa9Y0Y7/4QUe91t+QCgzqvdLnn3/uDtGjpi7n5u9L1eO3R4G+8847L5DLUfkdlbPRDzzqYgpoFlbg0d0c/yBQzAQIPBazDs/L7W7dut2WLV9pzZs1sbLep0Uq+/Zl2pfTvrasrCyb7AUm77zjZqtWNTXHl1m+YqVNmjzVWjRvZm1aR09EnOOKg07Ytu1o+/1Ny5atsNfffNduuuG31rqArulfi0cEEEhcAeUUuvPOO11uIb+VCgq++OKLOQoMbtmyxe655x73+zA88Pjxxx/bU089ZcGrMSr4+Oyzz1qbNm38y/KIAAIIIIAAAggkpYCmTAeXAwcOBF4qf6Gf/1EbmzZtGvFDXr3/8gOPynOoD2tzs3L1HXfcEbh2pCf6gFl5HbXIjcrWrVsjHcY2BBDIJwECj/kEWZyquf7666x+vbqBW9bKZQoajhg5xgYOesH++vjD2fJVBA6O8mTDhk1uz403XOf951I7ylH5u/nQocO2a9duywpKbJy/V6A2BBBIdAG9EX700UdNuYC0kqCCgNOnT7eHH37YfQ0aNCiuW1A9Dz30kHsDG/7GW4HNJ554wlq1amV//OMf3SqPkydPdlOFHnjgARs3blxc1+AgBBBAAAEEEEAgUQWCA41qY0pKSqCpu3btcu+1/A0K/GkBmvCi3JDBRSMXcxN4DK5DAcw1a9a4qd8ahem3UwNn/KL3cRQEECg4AQKPBWdbbGpWgtzzzzvX9uxJt/GfT7JZs+faad27htz/+g0bvV/466yUlzi4VasTrFatX4bia59GI6ps2LjJJfetW7dO4Py0tD32/dLlbhRRyxNauKnduqZfdP7ePXutXbvQvBxr1/7ogoqtWrbwDw08btmy1dasXede67gKFVLcaMvAATxBAIFiIfDJJ5+46c9KYt6lSxd3z3369LE5c+a4PERaqTA8EXkkmFdffdVN0znjjDNCPtHXsVqdUW9ulYeoa9ejvxuvuOIKL3XFPNNISI2UrFv3lw9zItXPNgQQQAABBBBAIJEFlDsxuFStWjXwUqtEBxctMqOvYxUFHnNbFOz87LPP7JtvvglZqCa39XEeAgjkXoDAY+7tODNMoOe5Z7nA47dz5gcCj/rUatj7o7zp2DMCq5Ad8kYY9j7/XLvqyr4u18YHH3xoy1escrW9/Mqb1v3ULnajN/1Znzy9+J8htnDhd266o1b50qikqt5U7gfuv8v7Q/1ocPLzCVO8HB0L7cXn/xnSoo9Gj7NdXmLiAU88ErJdL6bPmGk6T2Xsx5/ZUi+w+eCf7nWv+QcBBIqPgPIxapVD5RUKLq1bt3YfguhN8dVXXx28K9tz5W186aWXbPDgwTZhwoRs+xs2bOi2bdp0dGS3f4BeV6tWzWrXLpxR3v51eUQAAQQQQAABBPJbIDzwqPc4fvFHGfqv433M7Xn6m3HgwIEhC8wo56PyO1asWNH9Dbpo0aKQnJPxtonjEEAg5wIEHnNuxhlRBKpXr+ZGDgb/p/PpZxNd0LHvZRdZjx5nWoqX+2PylC9t+IjRLq9Gnwt6ucVeNFJy5KixNuifT1mlShXdFbQQjIKOClD2OOdML/hY1ltxbIX905vOPXHSVOv32/8XpSXH3nzlFZdZ61YtbfDzL9k9d99mJ7Vve+yTOAIBBIqcwLp169xowwoVKoTcW5MmTdzrbdu2hWwPf5GRkeGmWPurJUYKPHbv3t3lMlJwUm+ETzjhBJs0aZIbGXnzzTfnODVFeBt4jQACCCCAAAIIHG+BhQt/WYRUC8IoxYxf/IVe/NcXXnihez/kv472mNsZIW+99VYg6KiVrDXrpF27diGX+d///V8CjyEivECg4AQIPBacbbGsWUPq9Ye1yn4vofD4zydb8+ZN7eKLLgh49D6/p5uOPX3GN6bAY7RSoWIFu7DP+W50pD+1uk2bVlazRnUv0fCuaKexHQEEEIhbQKsYBk8F8k/0A4+7vVHTscqAAQNMOR2VqzFaKemlmPjHP/5h1113nb3wwguBwzQSUtsoCCCAAAIIIIBAMgusWrXKvvvuu8AtnHTSSW5kob9BIw0VjAzO4di+fXt/d74+av2B5cuXB+q84IILsgUdAzujPClVqlTInuDVuEN28AIBBOISIPAYFxMHxSuQdfCgNWhQ3x2+edMWl9esWrWq9tX0b0Kq0MjHdevWu+nU+qM8UmnXto3pS/kYV6z8wQUbf/hhje3wgo7+NOtI57ENAQQQiFdAbyR/+umnbIf7H3YoxUO0MnbsWDdy8d133425+vX8+fPtD3/4g1u98aqrrnIL2Hz55Zc2cuRIN437o48+ciPAo12H7QgggAACCCCAQKIKaEXod955J9A8l////PMDr/VEf+/Vr1/ftOCeivIu9u7d25stFzrjRPv0vkwzUpo3b66XOS6afRccKAxf9E8VKvVX8OIy4RepXr16yKYVK1ZY27ahM+QURN2/f3/IcXl5wQI3edHj3EQXIPCY6D2URO3TIjA7vVWiu3bt7Fqt5yqrVq22Des3uufB/9SpXcutKhbpPxwdp5Wu3/9glC1bvtJSq1SxuvXq2AktmtnWrbGnPvrXiPWfiX8MjwggULwFatSoYRs3Zv/95CdBD85PFCyl3y9PP/206dN6rcror8yoN8p6wzxq1CiXv7Fnz542fPhw05TsIUOGBN60du7c2Y2U/M9//mOjR4+23/3ud8HV8xwBBBBAAAEEEEhYAQXutACfAnL6IDY40Kdp1C1atMjW9ksvvTQw80OrS+t90Q033GCVK1cOHKsVqJWaRiMota9bt26BffE+0Xs3BTr9QN706dPtnHPOcSMuVYfa+u9//zswS0/b/GP1XEWBRwVQNXpSRdPIe/XqFWirjn/llVdM95GXEnzvMlX+bwVoKQgUNQECj0WtR4/j/cyc9a375dy0aWPXimreIjAq53q5HS++qI97npN/Xn/zHRd8/NMD93j5GE8InDpr1pzAc//Jf/9P8F+6x63eStkVvenaFAQQQCCaQIMGDbzcscts7969IaMO/U/kW7ZsGfFUfcqtAKMWp9GXX/SmUW9G//a3v7kgowKPWnxGK2OHf1KuqT8KPOqNOwUBBBBAAAEEEEhkgalTp7qRimpjZmZmtmCdtp922ml2ySWX6Gm2ounXXbp0sTlzjv4tpw9t//KXv5gW9EtNTbX169e7kY56L6WigGbHjh1jzirJdhFvgxYNVJ1Lly51uxXM+9Of/mS6vgKbK1euDAmU6iB9QBxcNC28U6dONm/ePLdZdTz00EOuDt27PmjO7cI3wdcJX2BQqXnUdqUBuuaaa4IP5TkCSS0QeY5rUt8SjT8eAgsWLrZPPplgjRo2sPbtjg5Dr1+/rvtkadGiJdmaNNBbIOYfA5/Ptt3foE+XNOLxxBNbhwQdNc1aX8GlamoVN1Q+I2NfYPOmzVu8QMLRXJOBjTxBAAEEwgTOO+88t2XKlKOr3Pu7P/vsM/dGVwvDRCoaqf3tt99m+7r88svdSEbtGzp0qDtVwU0FF/UmNbjMnDnTvWzWrFnwZp4jgAACCCCAAAIJJ6CAoAJ0+gofIaj3RZq9cf3118dcNE+5rRV89IsCgQruffHFF26Uox90bNq0qT344IM5Djr69f7+978PyTGpYOGsWbNcHkqNeExJSXGjGv3jd+zY4T8NPPbt29cFMf0N+tBZ6XOUP1JBR62OrZkzeSkdOnQIaYfaqWusXr06L9VyLgIJJ8CIx4TrksRv0FdffR0YZp6VdcjlYJw7b4FV8YbJ333Xrd5/EEdzouk/oD4XnGeffPq5fTZ+kp1+WjcvQHjIPp8w2ZZ6q1PffGO/qDeroe3K47hyxSpb8v0yF9Bc5eV3HDHyI1POtX3eL+XMzP1uFW1/hOX7wz+088/rYTu277SPxoxzK2hHvYC3I9ULWKosWLDY+0+jutWvV9e95h8EECg+Apo2oze3AwcOdL9b2rRpY+PGjTOtTt2/f3/3xtTX6NOnj2lBGE0NyknRitezZ892b6CvvvpqO+WUU9yb3zfeeMO0yqPqpSCAAAIIIIAAAskioL/zNKVZ72P0vkYpZDTS8FhF591yyy0u+Kj3W1u2bAksOKNRho0aNXKrYV988cXufdmx6ou2X2177LHHXD7tuXPnBqZMawq2RhT269fP9AGwRlWqKC/ktm3bLHgEYp06deyRRx5xHySvWbMmcCm1069jxIgR3joEOwP7cvpEZrfffrvpPSEzYHKqx/HJJFDCG1l2NHHBMVqtKWUbNmxwUf1KlSod42h2F0WBYe+PsslTvgy5tdKlS1mzZk3tRG+16VO7nWJ16tQO2a/vm49Gj/OCjVMCv/CVr/HSSy+0c84+I3Ds+M8n2chRY+3ZQc9YpUoV3fbVq9fae8NG2tp1R5MQV/I+VfrNtVfaXm/V7OEjRlvLli3sT/9zt/vE7c2h79nX38x252mK95lnnmY7dux05w544hG3/QMvMDlx0lQb8spz7rW+9Z8d/B/7fulyt/L2w/3/6LbH889zL75mb+1taxtKN4jncI5BAIHjJLD0roZWLSX24H4lRb/11lvNf1NZrlw5U7BQgcfgoik3ClJqMZho5a9//asLXCrQGFyGDRtmL774opvS7W9X0vSnnnoqxyst+ueHP+o/89UbdluN1BSrWrl8+G5eF5JA5gEv79WODGtUp4qVLRO6KmYhNYHLeAI70jItY/8ha1rvaNoXUBBAAAEEEk9AIycVfFTubH24q6BefheNotQ19HepZqHEEyANb4MWkVFgUO8R69ata+GrXocfn9PX+rtUAcxdu3a5adYK6EZbgDWndXM8AgUpoDynSlnVuPHRdHvRrkXgMZoM2/NVQNOgf1y/wUp5nzI1bdokMCoynots3rzVC1r+7OVIq+uS/OocLWRTomQJt+iMX4dGQO7yFrRp0KBe4Dh/X6zH9PQMK12m9DFHSAbXQeAxWIPnCCSuQDyBR7/1yi2kN5W5ySfk1xHrUdNnlFdIbyz1ib4Cj/n5xpXAYyz9wttH4LHwrGNdicBjLB32IYAAAggggAACeReIN/CY/x8p5L3t1FAEBTSKsa2XrzE3pZ63mnV4qfrfhWuCt1eokOKmXgdvi+d55cqM4I3HiWMQKOoCCgbqq6CKphcpqElBAAEEEEAAAQQQQAABBIqLQOz5Z8VFgftEAAEEEEAAAQQQQAABBBBAAAEEEEAAgXwVIPCYr5xUhgACCCCAAAIIIIAAAggggAACCCCAAAISIPDI9wECCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECDwyPcAAggggAACCCCAAAIIIIAAAggggAACCOS7AIHHfCelQgQQQAABBBBAAAEEEEAAAQQQQAABBBAg8Mj3AAIIIIAAAggggAACCCCAAAIIIIAAAgjkuwCBx3wnpUIEEEAAAQQQQAABBBBAAAEEEEAAAQQQIPDI9wACCCCAAAIIIIAAAggggAACCCCAAAII5LsAgcd8J6VCBBBAAAEEEEAAAQQQQAABBBBAAAEEECgNAQII5E6g/uFNdsv57XJ3Mmflm8D+g4ds3/5DViO1gpUokW/VUlEOBfZkHHRnpFYql8MzORwBBBBAAAEEEEAAAQQQQKCoChB4LKo9y30VuED7rGV2W5crCvw6XCC2QFr6Adu5Z781b1CFwGNsqgLdu2l7hlf/Eatfq3KBXofKEUAAAQQQQAABBBBAAAEEkkeAqdbJ01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgeAQKPydNXtBQBBBBAAAEEEEAAAQQQQAABBBBAAIGkESDwmDRdRUMRQAABBBBAAAEEEEAAAQQQQAABBBBIHgECj8nTV7QUAQQQQAABBBBAAAEEEEAAAQQQQACBpBEg8Jg0XUVDEUAAAQQQQAABBBBAAAEEEEAAAQQQSB4BAo/J01e0FAEEEEAAAQQQQAABBBBAAAEEEEAAgaQRIPCYNF1FQxFAAAEEEEAAAQQQQAABBBBAAAEEEEgegdLJ01RaikDhCuzavdumTp0R8aJbNm+2rKysiPvYiAACCCCAAAIIIIAAAggggAACCCBgRuCR7wIEYghMnjI14t5Dhw5b9erVIu5jY+EKlCpV0mpXr1i4F+Vq2QQqVyybbRsbCl+An4XCN490RfohkkrhbitXtrTpi4IAAggggIAvsH79evvkk0/stttusxIlSvibYz4eOHDAZsyYYfPnz7eePXta586dYx4fz84lS5bYggUL7LrrrovncI5BIOkFeEeW9F3IDRSUQPVq1ezF5/8ZtfpDhw5F3ceOwhOoXIGAV+FpR78S/RDdprD26O0z/VBY2tGvU6F8meg72VNoAvwsFBo1F0IAgSIs8OOPP5oCb61atTrud5kfbUlPT3cBxJzczBdffGHTp0+3Sy65xBo3bpyTU6Meu337dlPw0S8//fSTzZs3z9q2bWsVKzKgwnfhsegIkOOx6PQld1LIAmXK8MdlIZNzOQQQQAABBBBAAAEEECgkgW+++cbGjx9fSFeLfZnj1Za5c+fa2Wefbd27d7eaNWvGbmQu9x48eNCGDBliW7duzWUNnIZAYgsw4jGx+4fWIYAAAggggAACCCCAAAIIIFCoApqS/MMPP9i+ffts9OjR1rt3b6tQoYJrw+rVq23hwoX2888/28knn2zNmzcPTF3+/vvvbbeXK18j9xYtWmRXXnmlO2/nzp321VdfWWZmpjsnIyPDKlWq5Eb5+Te2du1aN/Lv8OHD1qVLF1ev9kVri6Y/lyxZ0jp27OhXke1x1qxZtmLFCqtRo4bVrl07236N6FRwUfdUr149O+ecc0wDTBQM/Oyzz0yjE3X+/v37rW/fvoHzdY62V/NmyXXq1Mnq1KkT2Dd16lSrX79+YKSoRlpOnjw5xNA/WMYa7agybdo0S0tLy5fp3H79PCKQCAKMeEyEXqANCCCAAAIIIIAAAggggAACCCSIQPny5a106dJWqlQp03MF+FQUyBs8eLALOmr/v//9bxdQ9JutYNzw4cNt1KhRduTIEbdZwbSnn37anauA5Mcff2zvvfeeC9z55ymI+K9//csF/BTsHDRokC1evNjtjtYWBTmXLVvmV5HtUQHTN954w9Wp4N+wYcNCjlHqLN3Ll19+aVWrVnX3oeuq3bpfXVdFgUj/uV6/++679uGHH1pqaqrt2LHDHn/8cdvsLT7qF+WEXLVqlf/SFGRVEFNBzvAiQ7/ucuXKuWuFH8NrBJJdgBGPyd6DtB8BBBBAAAEEEEAAAQQQQACBfBTo1auXC6pp+m+fPn0CNSt41q9fPzciURsVNFMeRE1H9otGRj7yyCOBgNqnn37qAnkDBgxwwcyLLrrInnjiCf9wU45DBQg1OtKvR0E4jXQ86aSTLFpbYi3OomDfxIkT7ZprrrEePXq4azVs2NDeeuutwHUnTZpku3btsmeeeca1T9fu37+/KaDZrl07d9+6t/bt29uZZ57pzlNQUvd86623WpMmTdw21aHgpa6V06I6atWqZePGjbNTTz01MMozp/VwPAKJLEDgMZF7h7YhgAACCCCAAAIIIIAAAgggkCACmoqsEYmaHrxmzRrbtGmTaRp1cNG0Zn8Un8LhaiQAAB+XSURBVLZrYRgtUKPRfSp6rFy5snuufzZs2OBGJe7Zs8cFC7UtKyvL1a3nuSlql6Zsa8EWv2hadHDR9Orq1au7adD+9ipVqtjGjRtd4NHfFvyo1bCvuOIKF5TVyEZdR+32R4QGH8tzBBA4KkDgke8EBBBAAAEEEEAAAQQQQAABBBA4poBGCWoEY7du3UwjCDVaceXKlTHPU77Epk2bRj1GIwYV0FNuSD2qaIp3165d3ZTu3AT1NDJTdYUHG4MboYCp6t6yZUtgs0ZYagRitKIRjy+//LILOHbu3NkaNWrkApXRjmc7Agh4HzaAgAACCCCAAAIIIIAAAggggAACCBxLYMKECXbBBRe4Lx07Z84cmz59eszTtBp0cM7D8IMV6FNAT1ONW7duHb47V691TdWpxVvatGnj6tDr4KLravSmpo7HW7Zt22bKR/nwww9b0/8GUzVyUsFTv6SkpLjFaPzXPCJQ3AVYXKa4fwdw/wgggAACCCCAAAIIIIAAAgiECWi0oAJqWoTFD9pptWZNsVYOxfXr17vchNqvFa6jle7du7tzlC9Rox8VrFQdfmnQoIHLbTh27Fhbt26dmyKtacxapMYvkdqigJ+Oj1TUTuVPVJ5Ifzq0gqbBRTkdFRD126URkFosR3ktoxUtQqMclkuXLnWLxWh175kzZ7r78s9RTkgtjCO7vXv3ujb4+yI9alq6gpUKavrOkY5jGwLJKkDgMVl7jnYjgAACCCCAAAIIIIAAAgggUEACnTp1ckHHu+++OxCM69u3r8tp+MADD9izzz7rciEq6Lh9+/aordD05QsvvNBGjhxp9957r02dOjVkOrOmRGuxlrJly7qFXu677z4XDDzrrLMCdUZqy/jx4wM5IQMHBj256aabXID0r3/9q/35z3+2unXrBu0113YtUKNVtu+//3578sknTQHL2rVrhxwX/EKL3lx11VWmwKjuZcyYMdahQ4eQ6dqahq46HnvsMTcyUvcVq2i6dw9vARwtfPPaa6/FOpR9CCSlQAkvoh463jjKbSh3g5K+KlFspUqVohzFZgQQQAABBBBAAAEEEEAAAQQQKAoCChdodGPwYjC6r8zMTDdKz8/JeKx7VXBSX1o0pkyZMqbA5eWXX25arCa4aL+OC16cxt8f3hY/lHGsNqj9FStWDOSP9OsLfkxPT3dxjmPVFXyOpmmr3mhl//79pkBlvDkqdbxs/EV4otXLdgQSRSAtLc2N6m3cuHHMJpHjMSYPOxFAAAEEEEAAAQQQQAABBBAongIKxIUHHSWh6cbxFq1qPXToUDdSUOdpxKOKRgqGl1ijA8PbEm+QMJ6BU5HuMbxt4a9jBR11rKZP56Tk9Pic1M2xCBxPAQKPx1OfayOAAAIIIIAAAggggAACCCBQhAU0GkojGzUtWVOyNZ1Z05RjrThdhDm4NQSKnQBTrYtdl3PDeRXQ0P+Vq1a76QUtT2jhDcmPPrw+r9cq7ufv2bPXli9faXXq1rZGDRvEPU3BpYbYuMlLNr3em95Q1ho1amj164XmdCnutjm5/02bNnupNjZZs2ZNvHw8NXNyauDYHTt32dy5C6x79y6WWqVKYDtP4hfISz8okft33y21TG8KT8MG9V1fxn9ljgwWyG0/7PGSy6/y/u/Yuzfdanppa9q0aemmUwXXzfP8E9D0u5mz5njOraxa1dT8q5iaEEAAAQQQQAABBJwAU635RkCgAAQUOHn7nQ8sw8vn4ZcL+5xvV1x+if+Sx3wQOHz4sD33/Mu2dNmKwMpu1apVtQcfuOeYgS/9cT9w0Ive6nWbXR4XP/fLuT3Ost9ce2XM3C750PQiVcXWbdtt8HMveSvs/ZIsXH/E33P3bVbWyz8Tb1Gw/pVX3rDVa9ZZ61YnEHiMF+6/x+W1H0aOGmOTp0zzfpZ+tp9++tn9TJ1+Wje74frr+HnIQV/kpR++/ma2vfveiJAVL2vWqG533nGzNW7cMAet4NB4BfSh1Wuvv2333n07gcd40TgOAQQQQAABBBAoAIFSj3slnnr1x7uWgldOhlh5F+Kpi2MQSEaBtLQ99qwXhKlXv67df9+ddsnFF3oj8ErYZ+MnuVGPGg1GyR+Bj0aPcyNVftfvGrv5pt9Zp5M72Jw5823mzG/tzDNPi5lwedCgF7xV5bZ6K+Ndb9f//jfWsUN7U9/N/nauN52jqjVp0ih/GlnEa9Hv/OdffMWtWnjvPbfbtddc4UadTpz0hW3evNW6dukUt8DoMZ94/vPc8WefdbpVZfRR3HZ57Yfvliy1d94dbr+6sLcLGJ9++qmWkZ7hfr4aNKjnTXWqF3dbivOBeekHBe71f0fDhvXtFu/32dVX/dpb6bKWfev9Tlvy/TLr1TM0qX5xds7rvaufdu9Os8Xe6N733h9pStLf/dQuVqdOrbxWzfkIIIAAAggggAACYQIHDhxwH6ynpsaeXVIy7DxeIoBAFIGPvOCJ/oi547YbrW7dOl6S5Up2+a8vsXr16tjESVOjnMXmnArs2LHTxn8+2c45+ww7ywsyppQvb829oO7VV/Y1Tdedv2BR1CozM/fbmrU/Wu/eveyUzie7VeRatGhm13ojHVUUuKTEJ/CNZ/XDD2vsN9dc6UYpqh+6de1sChzOm7/Qdu3aHVdFGnWk4Pxp3bvGdTwHhQrkpR800vStt9+3tie2tr6X/cp9aFjHC3j9/nfXeukHGrgAfejVeBVNIC/98P3S5aZR3Bf/6gJr3bql+6BKP0f6eVJQMt6fpWhtY/svAosXL7EHH/qLvfLqm7bT+/+CggACCCCAAAIIIHD8BQg8Hv8+oAVJIvCjly+wefOmlpr6S346raSm3IPbt+/gj/h86sf1Gza6qaCdO3UIqVF5GlUWeX9YRitr1q5zU0dbeP0UXGrVrOGN1k6xjIxfpsgH7+d5doEff9zgjSwt5a022C5kpwJWGlUUqx/8E/bty7Qh3lTH3uef6/Ks+dt5jF8gL/2w3QviK6ilUcLBpVy5cvaXx/rbxRf1Cd7M8xgCeekHPy/qzl2hgTB9yKIPsDQSm5I/Ai1btrBHH3nAfV16yYVHKy2RP3VTCwIIIIAAAggggEDuBFjVOndunFXMBBRoUX6vU7udku3O69Sp7bbt9qbzaiQkJW8CW7ducxXUC1sMplatGm5xmTRvGl200q5tG3vlpWez7VZuTo2GPLnjSdn2sSGygPpBKw0qSBVc/O93TV8/Vhn61ntudFffyy62WbPnHOtw9kcQyEs/+Lk5FXhXfsFly1dYVtYh9wFK38suMo1+pMQnkJd+0IjTut4CWR+PG+9+D2mK+9x5C23Fyh/sol/1Js9mfF0Q11EpKSnWtEljd+zGjZvjOoeDEEAAAQQQQAABBApWgMBjwfpSexERUM6orKwsq1ixQrY78v94T/fyplHyLrBly9HAY7h1qVKlrIa3GEN6Rs6cZ8+ea297Oe5U36/7Xpz3BhaTGrZs3eqZZV+xPd7v92nTvnarKD/66J/cyMliwpbvt5mXfvADj2+8+a7tTU+3rt7UXq34PmvWXFu0aIk36vFBl2sw3xtdBCvMSz9oZPxtt95gTz090JTv1C/6MIX8jr4GjwgggAACCCCAAAJFVYDAY1HtWe4rXwUOHTrk6lPOtGzF+6NSpUwZfpyy2eRiQ1bA+ki2s/UHfJnS8a2mrKDL8BGjbcHCxS5H5E3eog5MacxGGnWDRsZVSMneB97wLHdOmRirWmvxmfc/GGVXeXk564eNXI16QXZEFMhLP+zceTQP50/e762B/3jSFLxX6db1FPvnwOdNeWtvu+V6t41/YgvkpR9WrVptg59/2WrXqmnnnHOmNfbSFSz0Ar/TvpphTwz4uw144n/dwn2xW8BeBBBAAAEEEEAAAQSSU4BISXL2G60uZIEqVSq7KyonV3hRHjuVSpUqhe/idS4EUoOstQpscNm3b5/VqF4teFPE51O+mOaCjhrlqIU0zjyjO9MZI0pF36jveeUIDC/qA5VKlbKPhvSPHeYFHcunlHeBrmlffe02a6EaFS1M8+P6Da5PSpYkzbBDifFPXvqhcuWjfXSGt5K1H3TUpVq3OsHlqlXeQkp8Annph6lfTncLkz1w/13WpEkjd0HlIixbtoyN/fgzmz5jlsuDGl9LOAoBBBBAAAEEEEAAgeQSIPCYXP1Fa4+TgPJGVahQwbZ5i8iEF42sUwBFq1tT8i5Q08tHpyLr4MBjhhfwUp7G4G2RrqY/5PV1hhdsvOb/Xe5WxY50HNtiCygv4Pr1Gz3zzJDRWNu2Hf0ZaNggNCgcXNvBgwftwP4DbtSjv/3nI0dHC0+Y+IWV9EZNdvdWuS5L4NHnifqYl36oUb26q7d62OIlGjmsafQ/eSstU+ITyEs/rPKC7tW9D0z8oKN/xa5dOrvfVaxq7YvwiAACCCCAAAIIIFAUBRhuUhR7lXsqEIFTOnd0K1f7OQh1EU29njtvgbVo0cwqRciHVyANKeKVdjq5gxuduGDBopA7/fbbeW415Y4d2odsD36xZ89et4BD+3Yn2g2//w1Bx2CcHD7v3Plkd8b8+aH9oJyZmmZ9ordgRrTycP8/2r9fHBjydeP1v3WHP/TgfW572RhTtaPVWxy356Ufmnu/lzTScfac+SF027fvtE2bNluTpkcX4QjZyYuIAnnpB32YouCiv3CWf4Hvly53T7XwDAUBBBBAAIFkEpg1a5aNGTOmUJu8fv16e+mll9zfA7m98O7du10d6enpUavIzb3pb8JNmza5r4ipucKupjReO3fuDPlS2ygIFFUBRjwW1Z7lvvJdQIsAzP52rr362lC74vJL3WIlGlmXkbHP7rzj5ny/XnGtUCODTj21i339zWzTCsoK+K5es85GjhpjnTt1sNatWwZoXhky1FauWGWP/+Vh1x9aJVYrkOsP/U8/mxg4zn+i6cFnn3W6/5LHGAKyVkBk+MjRVtrLX9q4USObOetbmzN3vhtJWq5c2cDZ4f0Q2MGTPAvkpB/6P/y41fLyCGpKr0pNbzEmfb9/OW2Gjf98kvez1Mky92faiJFjXEDy0ov75Ll9xaWCvPSD+mDZshX2yqtDvRyPZ1irlifY0mXLXZ+kVqni5dzsXFwYuU8EEEAAgSQU0MJ08+bNs7Zt2wYWHty8ebOtWrWqUO9GwcL580M/TM1pAw4cOODquOqqq6KemtN727Jliz3//POm9unvkNTUVLvrrru899F1o15j7ty59sYbb4Tsr1atmv3tb38L2cYLBIqKAIHHotKT3EeBC2iK7x/uvMX+89LrNuhfL7rraZTj9V4OwebNmhT49YvTBa7/3W/s4IGDbgXYj0aPcyMglZfuphv7heRqzPBWEt+dtifwyedKL/CoopxqkYoWOiHwGEkm+7ayZcva//zxLhvofa+/OuQtd4BGOioAf16vHiEnhPdDyE5e5EkgJ/2we3ealStXLuR6V1/V18xbD2jkqLHuSzsV7Lr3ntu9N8SkhwjBivEiL/2gwGJGRob3++xTe+vt9wNXUXoO/U5TGg8KAggggAACiSqgFDpDhgyx/v37W/PmzRO1mcetXcOHD/cGS9RxPmrE0KFDbcSIEXb33XdHbdOePXusVatWdscddwSOIfd5gIInRVCghBeVj7BsafY71ScdGzZssBo1arCIRnYethQjAQ2fV+47PTb1pioqXxqlYAQ0mvTHH9db48aNYi5mUjBXp1ZfYLuXb3Pnzl2mqbtMkfZVCv8xL/2gn6X1GzZalcqV3UjW4MVmCv9OkvuKue0H/eG2YeMm27s33Y1M1Qch/JGR3N8LtB4BBBAo6gI//PCDG+04adIkO+2006xDhw7WuXNnGz16tBvxeOWVV9rs2bOtivehZvfu3a1q1aqOZNu2bfbNN9+4Y6dPn+72NWvWzO1bu3atq/Owl2u6S5cuIcFMxRyWLVtmixcvtvr161vHjh3dCEKd+P3339vgwYPdqMCvvvrKsrKyvFlSp1ojb1ZOcNmxY4ctWLDAS5G1xfsborGX9/2MwCJ7Gs34+OOP29NPP+3iGjpP056/+OILd7zauH37dlu9erU98MADrlqNstT/12pLeNG5CjDecMMNri3av2LFChs0aJC98MILVrp05HFe77//vlt4TudREEhmgbS0NO+97V73sxbrPsjxGEuHfQhEENB/PFokoJk3ypGgYwSgfNykqdFt27Yh6JiPprmpStN327RpRdAxN3j5eE5e+kE/Syd6fdigQb3Am+98bFqxqiq3/aDRqC2aNzPlsdXiTAQdi9W3DTeLAAIIJKWAAmfly5d3bdf/Y5r94peNGzfae++95/YvWrTInnzyycAsJAX/Pv30U5dPcZ+3QKQ/1klBvH/961+mD+O0XQE6BRn9MmrUKDdaUNOVlyxZYn/+85/dQof+fj0qoKeAnwIeuqba4ZetW7e6oOK3337rZhR89NFH7hrR8i6qXc8++6x98skn7j4U9FQQMrgo4KntkYraoDqaNm0a2F3dW9xP27QvWtGIR7Vp8uTJpnvWvVIQKMoCkUPwRfmOuTcEEEAAAQQQQAABBBBAAAEEEIgp0KRJE2+Ufi0bN26cG9EXPNU6JSXF7r//fhew69WrlxshqJGCLVq0CNSpqcQNGzZ0rzWaUXkNNUry7LPPdtsUzFTQ76STTnKBOI1k1ChAjapUefvtt23NmjXWrl0791r/aL9fp0Y1Lly40PtgtYHbr0Boy5YtA1OYzz//fHvwwQfd6EuNfAwvCnoqV6VGQdarV8/tfv31171F4XYFDr3uuusCz8OfZGZmuk3BaVOUnkVl//797jHSPxohptGXClAqWDphwgRT3snzzjsv0uFsQyDpBQg8Jn0XcgMIIIAAAggggAACCCCAAAIIFJ6ARvb5oyErennvNd1awbTgwKMfEFSrlLZNIx012m/ixKOLQGq6tFaDVtFMgBNPPNFGjhzppj2fcsop1q9fP7cv+J/gOhUsVPBRRSMIFaQMXjimspdi5oQTTnDBxUiBx3Xr1rnp4X7QUfXovoIDj9oWrcSa/RZr33333WcKxPp+yhM5duxY69mzJzMiomGzPakFmGqd1N1H4xFAAAEEEEAAAQQQQAABBBBIbAEF8xSM2717twsWKmConNNdu3Z1QUO1/vbbb7df//rXXj799TZgwAAbOHCgyx8Xz51phKECm7Vr1w45XNO2o40+1CrXWsMit0UBVxXV4xf/ub/P3x78qCnrftBR27t16+baHm/AM7guniOQDAKMeEyGXqKNCCCAAAIIIIAAAggggAACCCSpgKZsa2qxFoRp3bp1trvQ9GNN1daCMwpGKlio4OOUKVOsb9++2Y4P36BAn76UjzG4/uXLl9u5554bfrh7XbNmTZs6darLGennr1Qb4y1aTEfBU02X1v2paGEd5cZUwDNSUX7Kt956yy677DLT9VUUjFXR9HUKAkVRgBGPRbFXuScEEEAAAQQQQAABBBBAAAEE8iigkXkKiCmglpOgXPhlNUVaOSI1pVhTnLWq9YwZM0zTjP3y8ssvm1bQ1j6NXtT0aQX24i2aqqzVtJX3UfkXtfq2pnNr2nakou2a4j1ixAhLT093Iy1nzpwZcqiCoWpvpKK2KVCqe9IoTS10o3yYfr06RwHVuXPnBk5XgFPX1MI3uqZcFVzVFPVYoyQDFfAEgSQUYMRjEnYaTUYAAQQQQAABBBBAAAEEEECgoAUUJOvRo4cbpffdd9/ZzTffnKtLapr1rbfe6up55pln3KjAunXr2k033eTqU45ILUajBWIUMFSws0OHDta7d++4r9enTx8XcHz11Vdd0FK5G1VnnTp1Itaha95yyy325ptv2rRp01x+R+WEVB5Kv4wfP960YEy0+1ZOySFDhrjVtHWOztcCOn7RitXvv/++dezY0d2ztl966aU2ZswYt2q3RnYqIHvjjTf6p/CIQJETKOF9ahHXWGIlP1VCWOVAqFSpUpGD4IYQQAABBBBAAAEEEEAAAQQQQCC7gAJkGq2nacR5LRqFqNGMwXkOg+vct2+faaXoWAu0BB8f/lx1K9di8GrT4ccEv1ZIJCMjw7QYTXjxwyXHaotGaOrYSPek9iiAG140slNfkc4JP5bXCCSiQFpamhvV27hx45jNy/tvjZjVsxMBBBBAAAEEEEAAAQQQQAABBJJZID/zD2oEYayS1ynHCvLFG3RUOxRUjBR09PfFaqu/r1y5cv7TbI+Rgo46SEHc/AjkZrsgGxBIMIHsYfcEayDNQQABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BMg8Jh8fUaLEUAAAQQQQAABBBBAAAEEEEAAAQQQSHgBAo8J30U0EAEEEEAAAQQQQAABBBBAAAEEEEAAgeQTIPCYfH1GixFAAAEEEEAAAQQQQAABBBBAAAEEEEh4AQKPCd9FNBABBBBAAAEEEEAAAQQQQAABBBBAAIHkEyDwmHx9RosRQAABBBBAAAEEEEAAAQQQQAABBBBIeAECjwnfRTQQAQQQQAABBBBAAAEEEEAAAQQQQACB5BPIceDxyJEjyXeXtBgBBBBAAAEEEEAAAQQQQAABBBBAAAEEClUg7sBjyZJHDz106FChNpCLIYAAAggggAACCCCAAAIIIIAAAggggEDiCCg+WKpUqWM2KO7AY4kSJax8+fJ24MCBY1bKAQgggAACCCCAAAIIIIAAAggggAACCCBQ9AR+/vlnFx8sV67cMW8u7sCjaqpUqZIporl79+5jVswBCCCAAAIIIIAAAggggAACCCCAAAIIIFC0BHbu3GkKPlauXPmYN1b6mEcEHVCxYkXLysqyvXv32uHDhy01NdXKlCljGg1JQQABBBBAAAEEEEAAAQQQQAABBBBAAIGiJ6BAoz8Y8eDBg1atWjWLZ8RjCW+xmByvFqPAY1pamuXi1KInzx0hgAACCCCAAAIIIIAAAggggAACCCBQDASU11EDEeMZ7SiOXAUedaKCjvv373cjHwlASoSCAAIIIIAAAggggAACCCCAAAIIIIBA0RPQbGfNek5JScnRzeU68Jijq3AwAggggAACCCCAAAIIIIAAAggggAACCBQrgRwtLlOsZLhZBBBAAAEEEEAAAQQQQAABBBBAAAEEEMi1AIHHXNNxIgIIIIAAAggggAACCCCAAAIIIIAAAghEEyDwGE2G7QgggAACCCCAAAIIIIAAAggggAACCCCQawECj7mm40QEEEAAAQQQQAABBBBAAAEEEEAAAQQQiCZA4DGaDNsRQAABBBBAAAEEEEAAAQQQQAABBBBAINcCBB5zTceJCCCAAAIIIIAAAggggAACCCCAAAIIIBBNgMBjNBm2I4AAAggggAACCCCAAAIIIIAAAggggECuBQg85pqOExFAAAEEEEAAAQQQQAABBBBAAAEEEEAgmgCBx2gybEcAAQQQQAABBBBAAAEEEEAAAQQQQACBXAsQeMw1HScigAACCCCAAAIIIIAAAggggAACCCCAQDQBAo/RZNiOAAIIIIAAAggggAACCCCAAAIIIIAAArkWIPCYazpORAABBBBAAAEEEEAAAQQQQAABBBBAAIFoAgQeo8mwHQEEEEAAAQQQQAABBBBAAAEEEEAAAQRyLUDgMdd0nIgAAggggAACCCCAAAIIIIAAAggggAAC0QQIPEaTYTsCCCCAAAIIIIAAAggggAACCCCAAAII5FqAwGOu6TgRAQQQQAABBBBAAAEEEEAAAQQQQAABBKIJ/H//2UooOoOkHwAAAABJRU5ErkJggg==)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "markdown", - "source": [ - "![CleanShot 2023-09-26 at 18.38.50.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABSQAAAGPCAYAAABF+95DAAABXWlDQ1BJQ0MgUHJvZmlsZQAAKJF1kDtLQnEYxn+WYVeSaCkaHGoo7IIXaMyEInAQM7oQwel4i9QOR02CvkBbQwQhREM0BUGLjUFTEC1BQQiNfYDApeT0nqzMoj+8PL//w8PLywMNzYqmJa1AKp3VQ9OTjoXFJYftmRZ6sTNKh6JmNF8wGJAIX1r/yvdYTL0bNncdnIauzw5XYpu3R4X+0tPe33zda41EM6rom8yIqulZsDiFg/msZvK2cLcuRwnvmhyv8rHJq1W++MiEQ37hG2G7mlAiwiVh5+oPP/6DU8mc+nmDeX17ND03K9ol00eAKVx4CMt4GZMf/+Q9H3k/G2hsobNGnARZHPjE0UgSFZ4hjcoITmGX7HPhNnv+3V/Nyw3BhNn1es1b9sJ5D3Tu1LyBPHQU4GpcU3Tlu1VL2ZqJuV1VbitC075hvMyDbRAqD4bxWjSMygk0PsJl+R12bmMC5eqO1QAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAAFJKADAAQAAAABAAABjwAAAABBU0NJSQAAAFNjcmVlbnNob3Q/fkkfAAAB12lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zOTk8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MTMxNjwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgpzR3A3AABAAElEQVR4AeydB5wUNRvG36UjSEdAqgUERRELolRRQUQUFBVsYPlURLEXRMWGIIK9i72LIogiRRBEFBRBKQoKSFV67+X2y5O7d8jO7e7t7rW9u+f9/eZmJpPJJP9kZm+eeZMEgsaERgIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAI5QKBQDlyDlyABEiABEiABEiABEiABEiABEiABEiABEiABEiABS4CCJBsCCZAACZAACZAACZAACZAACZAACZAACZAACZBAjhGgIJljqHkhEiABEiABEiABEiABEiABEiABEiABEiABEiABCpJsAyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAjlGgIJkjqHmhUiABEiABEiABEiABEiABEiABEiABEiABEiABChIsg2QAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkGAEKkjmGmhciARIgARIgARIgARIgARIgARIgARIgARIgARKgIMk2QAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkkGMEKEjmGGpeiARIgARIgARIgARIgARIgARIgARIgARIgARIgIIk2wAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECOEaAgmWOoeSESIAESIAESIAESIAESIAESIAESIAESIAESIAEKkmwDJEACJEACJEACJEACJEACJEACJEACJEACJEACOUaAgmSOoeaFSIAESIAESIAESIAESIAESIAESIAESIAESIAEKEiyDZAACZAACZAACZAACZAACZAACZAACZAACZAACeQYAQqSOYaaFyIBEiABEiABEiABEiABEiABEiABEiABEiABEqAgyTZAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQYwSKJHqlXbt2yc6dO2Xv3r2JJsHzSIAESIAESIAESIAESIAESIAESIAESIAESIAE8gCBYsWKScmSJaV48eKZzm0gaCyeVBB93bp1smPHDgkEAoLMYE0jARIgARIgARIgARIgARIgARIgARIgARIgARLIfwRSUlKsUyJ0wYMPPlgqVKiQqULG7SG5du1agXckLowM0EiABEiABEiABEiABEiABEiABEiABEiABEiABPI/gU2bNsnmzZsFAmWlSpUSLnBcguTWrVttN21csFSpUglflCeSAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAnkLQLlypWzPaUhTKL7dqL6YFyT2kCQRD/xRC+WtxAztyRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAi6BsmXLSpEiRWTbtm1ucFzbMQuS2le8RIkScV2AkUmABEiABEiABEiABEiABEiABEiABEiABEiABPIPATgs7tmzJ+ECxSxI6tw3UEBpJEACJEACJEACJEACJEACJEACJEACJEACJEACBZMA9EHVChMhELMgmUjiPIcESIAESIAESIAESIAESIAESIAESIAESIAESIAEXAJ0d3RpcDvfEsiMap9vobBgJEACJEACJEACJEACJEACJEACJEACJJBGIBAI5BgLCpI5hpoXymoCFBmzmijTIwESIAESIAESIAESIAESIAESIAESKKgEclJnoSBZUFtZHix3Tt4YeRAPs0wCJEACJEACJEACJEACJEACJEACJEACOUIAGo0uuGC83pUUJHOkmniRzBCgEJkZejyXBEiABEiABEiABEiABEiABEiABEiABLKXgGo3sQqTFCSztz6YeoIEtCEneDpPIwESIAESIAESIAESIAESIAESIAESIAESyGECrp4TTZykIJnDFcPLRSfgNtzoMaMfzap0ol+FR0mABEiABEiABEiABEiABEiABEiABEggfxKIJijGUmLVZsKlQ0EyFoKMkyMEtKHGc7FEzoknfcYlARIgARIgARIgARIgARIgARIgARIggYJIIJLmEk5gjMYH6fjPoSAZjRiP5QiBSA3cf/FY4/nP4z4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEDWEIA+4y5I1S84+q+kmo7GoyDpJ8T9HCOgjTGjC8YaD+nEEzej6/I4CZAACZAACZAACZAACZAACZAACZAACRR0AioiRuOgekxGcTUeBcloNHksWwho44uWeFbFyew1op3PYyRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQFwlkJBy6ZYpFo9H4btxo16AgqcS4zhECbsOMdMFIcSKFazoZHdd4XJMACZAACZAACZAACZAACZAACZAACZBAQSYQr4biFxdxvi7g6D+OMBwPF45jFCRBgZYjBDJq7JGOxxseqTCR0okUn+EkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUBAIRBIOtewZaSp63J9OpHAKkkqW62wloA0w0kXCHY81DGmGixvpWgwnARIgARIgARIgARIgARIgARIgARIgARI4QCAWXcUVGzU+1uHC3TBcxR+PguQB9tzKJgLaSMMlH+6YP8y/j3TChWn60Y5pHK5JgARIgARIgARIgARIgARIgARIgARIgASiE3CFRVdvibSt8fW47uMqbhgFyejceTQbCWhDdC/hhrnbGscfpvu6jhRPw7kmARIgARIgARIgARIgARIgARIgARIgARLImADERFdvUXFR1ziGRfeRosbXMP9xjUNBMmP+jJEJAtoQ/Un4w6Pt45h73N1Guv59vVakcD3ONQmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHoCEBT9ugr2NTwlJUWwIEzj4Vg4IdI9rleiIKkkuM4xAtoQ9YLuvruN42jcau4xdxvH/fvhztEwrkmABEiABEiABEiABEiABEiABEiABEiABMITUNHRf9QNVx1G13oM+4UKFbKnuscQgH0VLClI+ulyP8sIaMOLlqAbx90OJ0S6x91tpO/f12tGCtfjXJMACZAACZAACZAACZAACZAACZAACZAACcRPAJqLX3eB4KiajitMqhCJ+NimIBk/b57hEQgaJdB4MJrGZNc23GxLwIshYlRxq4ynhrkNNdw2wjRc10hMt921brvHnQvbTTeO/xj3SYAESIAESIAESIAESIAESIAESIAESIAEQgmoeBgaatQeIySqQW/RBeHY1jXiYBvCJNYajjUMcSlIWhT8EzMBCJAp+03r2Ze2hiCZtthEUgVJNC7T4sxiBMlAYbsEsV3INDmsHdMGjCB7Xpi1quvh4rhh2FZzz9EwrkmABEiABEiABEiABEiABEiABEiABEiABKITUO/GSLGg30B3cTUdPccVIFXn0XRUlKQgqUS4jk4gaETI/XtTlxSIkWZB2IZ/JLhxici21SK7NtvjwYBpVsVLiZSqLIFytUUq10sTIo0wCUGycFGxccy223C1kbpr//GVm1bLmN+/l1+XzJP9RhjVuNEzz6MkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAJZQcCKikaQbFD5MLmkzHlSrazRfxyPSFzDL0rqvh4LGEEHLm0Z2v79+2XFihVSsWJFKV26dIbxGSGfEID3oxUid5v1HitEBhdPFlk6VYIrZ4rs2XqgoMUPNmJjMaNTGqFytwmHYAmDh+Shx4vUOlUCR5wuwaKm/RhRUooUlxSIl+a4NkNdq3cj9rGkmHwM+vp1eWXiR6lp8i8JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkECuE7i1XQ+5pW13K0JCeMQCcz0mse+GU5AEEVp4AinGI3LvLiNEGjHSrINzhknwz1EiO9eLlK4igWqNRMofJoHSh4iUKGPSKOQJizbBPdskuH2tyMblkrJmrvGmXGxanxEg63eQwDEXSEopc54RJYOFi6cKmWnio4qSKkZifd+wIfLRtK/C55OhJEACJEACJEACJEACJEACJEACJEACJEACuUbg2lYXS9/zeqYTJVWgVDHSWxuxhx6SuVZdSXxhK0LuFNlnhMi/xkrw13esEBkwXo5Ss4kEytYKm/lwzclrYdvXScrKGRJcON54PBpt8vjLzdJNgoVMF+4iJc3aeFempYp01EsS3bO7PH9T2OsxkARIgARIgARIgARIgARIgARIgARIgARIIPcJfH3763J09SNtRuAd6YmPaV6TKk4iQujsIrmfd+YgGQgYEVJ2bzNjQm6S4KSBEvzhKTMWZE0p1PJuCTTsElGMzDDrpSpJoG47CbTpJ4HDW0vwt/dl/+i7RNb/LUFzveDe7aabd4oVIlWMxPqlb9/PMGlGIAESIAESIAESIAESIAESIAESIAESIAESyD0CQyd96jmXQc9RbQdOZ+rApmt22c69ekrOK0OMNF2tZf0iSZk0wHS3/kcCjS6TQPUTMsyvNio3oucdaQJtAzQ+kBovZd0iCc4eJil7d0ig5Z3G87KppBQ2npJFSlhPSY13xF1nuElymwRIgARIgARIgARIgARIgARIgARIgARIIMkIFClUWOYPHBvSbRteka63pI4rSQ/JJKu8XM0OumlDjFy7QFLG9LGzZhdqfntMYmQi+Q5UPEICp/Uy41DWlpSJj8r+RZNMF/HtxlNypxUtoaTv22dm86aRAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkkNYF9Kfs9Pcc6paV5qUHfUaczXVOQTOqqzMHMYQIbdNPeslJSJjxsJpkpLIVO6SlSpnpMmdAG5UYO5x0Zetx4SxYtJYUaXylyyLESnPKk7F/+iwQxGQ7GrjQJYHZ3GgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQPITgI4DPUcXt9s2cq/hFCSTvy6zP4dm3EbZY8ZvNKJkyuQnjTC5VQqdeI3IQRWy/NpoeDBXrEzBUKbHXSzB8keY8SqfkeCW/yTF5CFl3x5vvIEszwgTJAESIAESIAESIAESIAESIAESIAESIAESyFIC0H0gQvq9IjVcL0ZBUkkk8XrHjh0ye/Zsb9m6dWvW5naf6aoNj8RfhoqsniOBE3qImAlostKMNh6SnCriGhgMFJbAMRcaTXS37J/2qsj+fcZTcodVzjUO1yRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAslLwBUiXWFSHdSQc2wXSdYiQISLZBgAs0SJEpEO57vw+fPnS5cuXbxyvfnmm9K6dWtvP1Mbpn+/YHbrNX9IcM4wCRx1jmBsx8QNwmMg6ukh3pEpkCrTvCZLlBUxs3gHZ5nZt+ePkeCRZ5ku29HTinohHiQBEiABEiABEiABEiABEiABEiABEiABEsgxAhAh1Qqb4QDVIQ2T22Abx6HrJa0gec4558iyZcu0DOnW5cqVkxo1akitWrWkW7du0qxZs3RxGBADAcyqnWK8EWd9YL0iA4e3juGk0CghKvdvH0ng2IvMGJRFvUgp6BJuTOOFW0OkRHigcn2Rao0kZfZnInVaSzAQWZj2LsANEiABEiABEiABEiABEiABEiABEiABEiCBXCcAwVHFR6vzpAmRqgXpsaQVJDMiuGnTJsEyd+5cGT16tDRp0kQGDRpkBcqMzs2Px0eMGCHr16+3RatZs6a0bds242JCKNxnZrT+9zcJrpwhgcZXGOfGwhmfFyFG8N9ZJp2ZRtg8ROTIMyPESg3WhqiRsI8lxXhMptRsJikrfpOU+V9LypHtNQrXJEACJEACJEACJEACJEACJEACJEACJEACSUzA9ZDUbMIjEpoPxEi1PCFIFilSRCpUODDBCgq3YcMG6+apBfn555+lZ8+eMnz4cClevLgGF5j10KFD5Y8//rDlbdOmTWyC5H4zs7bxjpS/xoiUrCCBascnzsukFTQCIiy4aIIEapwkUqJcuvRM+/MM27bDdlqYHgtC0KxyrKQsnCDBI2IQVr0UuUECJEACJEACJEACJEACJEACJEACJEACJJBbBPweksiHOqWpMxrC8sSkNvXr15dp06Z5C8THefPmyTvvvCN169ZFOaz9+eef8vTTT+su1xkR2G+6a+81E8cs/k4CtU7LKHbU48FFE0V2bkqNY8XJr8LGdxshImhjRIPFNrp327Cqx0nK5v9k/6p5YdNhIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHIRsNpOmsbjbqvugzBYnhAkw6GFF2SLFi2sKFm6dGkvyvfff+9tcyMaAdMA9u2R4LKfbaRAlaOjRY5+bOdGI2pOSk0nbUIcdAOXDf94Kni4BHQyGz2m3pK2cZY/TIKFikjKSpMOjQRIgARIgARIgARIgARIgARIgARIgARIIOkJ7N+/33M+g76ji5txhOWJLttupv3bVatWlZYtW9pxJHFs4cKFsm/fPkE3b7+hwJMnT5Zhw4bJokWLZNu2bVKvXj2BB2anTp3stv8cd3/16tWCGa7RNXrp0qVStGhRqV27tjRs2FCuuuoqKV++vBvdbn/11VcyYcIEu43ZhQYPHpwuDvLxwAMPeOEXXnihNG/e3NuPtDF9+nT5+OOP7eGVK1d60ZC/2267TUqVKiXt27cPP+EPZtcOmmXV7NSu1aWreOfHuxH803hDovs3un03uVaC04caMXKRpPw5QuTU3mZcytSZlMKlC2E8tXGmHtVtM5SkSKWjJGXNn+FOYxgJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkEASEoC243bdRhZT9Z5UgRJjSaZX7ZKwIBllqVq1al4UiJHr1q0TCJWubdy40Y4xie7erv37778yadIkeeutt6Rv375y+eWXu4e9bQiREBN37TLdnB37559/vPP79Okjl156qXNUrHg5cuRIGxZJkNy9e7doHERs3LhxTIIkZiF3z9MLr1q1yoZXqlRJGjRoEEGQNGNHGguu+0sCFY7UU+NfG+Ex+N/v9rzA0R2Nz21RKdTwPNk/5RmRzSsluPwX0x28iT0O8RGGRhnJ0EBhWKWUqW4m3Blj9g62YfxDAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQvARUjMRENjDVefw5zrNdtt2CLFmyxNuFZ2TlypW9fWzs2LFDunTpIq4YCTAHH3xA6IIo+OCDD9ou4CEnmx14OT722GMhYiSEvipVDngVbt++3Z4PcTOnDAJnyZIl7aIVjWtrODwkw3mK2vzBQxK2eYXR+0LF29QDMfyF4j3XeEEaC1SqK4Gqx9rtYOnqEqh5Sur239+YcSpDRVwcUHHSRrL7B9x4vXEkS1RMF0/jc00CJEACJEACJEACJEACJEACJEACJEACJJBcBNyxIt1tvzCZ5wXJOXPm2G7Yih/ehRDkXBswYIDAkxEG4e6OO+6Q2bNny++//y4TJ06Uk04yM0Kn2QsvvGAFTN3H+plnjLdfmqGLNibY+frrr+XHH3+Ujz76yJvVG6CfeOIJjZrt6wsuuMBO7oMJftDtXK1Vq1Y2/LvvvpNLLrlEg0PXZvIY2bPdjCO503S1Lhd6LMa94PJpEtzyr+2SHTj6fDOpzUbTVXuJyPY1EqjXXqToQSK7t0lw4fh0KaaOF6nekLo+EA0NNVC8tJmFm0YCJEACJEACJEACJEACJEACJEACJEACJJAXCKjwqGvkWbfddZ4QJJFheDnqsnXrVlmwYIG8/vrrcsUVV9gxI7VSevbsqZt2je7bOs4iAtq0aSO9evWSgw4yYpmxOnXqyEsvvWTHW8T++vXr5YMPPsCmtc2bN8vixYt1144Vecghh9h99Hk/5ZRTpHPnzt7xv//+O52g6R1Mqg0j9e3ZkZqjIsXjz9nenRJcYLwfjQVqN7NelsHl0yXlpxck+JfpZl2slBEl29rjwSVTjUi51m67f+Alqd23IVDCXM/JYKFiIfvuudwmARIgARIgARIgARIgARIgARIgARIgARJILgLQ8FR4RM50W9fqNZknxpCEByAmjsnIbr31VmndunVItNGjRwtm+FHzj/GIcIy12KxZMxk3bpyNNnfuXI0umMEbk9VgDErY+PHjrQgKMVLtkUcekfvuu093pUSJEt52cm+k+R86ZYk1v8G/xqZ6WBY9IDymO7fmqSJLfxLZukqCf44SOfHqdFEQoCKkNk5dywHEYc9jIAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQHISUH3HXauelicEyYywVq9eXe69917p0KFDuqjomq2mXpGYadtv7liLy5cv9w6j+/cZZ5whn332mQ2bOnWqtGvXTjp27GhFzEaNGtlxGiFc5i0zap96Ru4zM2THY9tWS3Cp8Xo0FjiqXWrX7HDnBwpJoMF5Evz5NQmumS+CpfJR4WJ6irketCJlyj522VYgXJMACZAACZAACZAACZAACZAACZAACZBAHiGgIqRm17+fJwRJiIXwYlSDe+eaNWt0V84999ywYiQirF692ouHLt9XXx3eS8+LZDZcQRLh999/v525WyesWbhwoTz99NN2gRDZvHlz6dq1q7Rs2dJNJrm34RVZoqxRFM14m7u3xpXX4DwzkQ3GoDz4UDODtvGCjGKY7EaqHCvBVXMkOP9LM/nN7SZ2ZNdH7bqNJPeb8SdpJEACJEACJEACJEACJEACJEACJEACJEACeYMAhEfodjq/i1+I1FLkiTEkMWELJpDR5aeffgrpwv3222/LqlWrtEwha8x+Ha/5zylTpowMHTpU+vbtGzJ5DNLdtm2bjBkzRnr06CH33HNPyHiW8V43R+MXSpv4p0w1Mw35AdE2ozwEV8+T4Lq/bLTAMWYiG6e7N2bWLnTqjRKoa7wmHQs06CDBQkb73r5O7HiSzrGomzs3pfOcjBqfB0mABEiABEiABEiABEiABEiABEiABEiABJKeQJ7wkPRTRH/z22+/3fN23L17t/VWDDfDdc2aNeW3336zSaBr96OPPupPLt0+ZuL2G8KuueYauyxbtsyON/nDDz8IxNG9e1O7PA8bNkz27Nlj8+I/P+n2IRAaC1Q4UoLrF0bxWXRynrLfjAX5Zep5VY+TQEVzrg4AidCS5VOXtKEpvTMPqiiBw1pIysLvJPj3eAlUO9508y7lHQ5Jw4QiSespufU/kdJmAqH1u7y43CABEiABEiABEiABEiABEiABEiABEiABEkhuAtB60us9BwSjPClIAjkmrznxxBPl119/tTUwfPhwufbaa6VuXdNF2DF3H16UmBW7ZMmSToz4N2vVqmWvheuh6/hll10mixYtsgmNHTvWipLFihWz++7YlKgIiKfFixeP/6JZfUbAVL0Z41GqHCPyzyTTbXuLSPEyUa8S/MeMvWm8HKVQUQkcfV66uME1f4qs/UuCZatJoHqTkOOBI9qYvvAzbPdwOwv3MReGHA+3E1xvmFasbybGOTDJULh4DCMBEiABEiABEiABEiABEiABEiABEiABEsg7BNK7AuadvMudd97p5RYzaQ8ePNjb140GDRropp1t+9133/X2/RuYXRvjTLrWv39/ady4sV0ggu7cudM9LIcccoj07t3bC9u1a5fMn28mb0mzQw89VDdtH/qff/7Z29eNDRs26GbCa52lCAm4s4pHTBBdrQsbYbHGSTZKcO3fEaPaA2acyeDCCXYzcHjrVE9Iu+f82bTUdMmekjp5jRNsFfHCxSVQP3XSoeBKIyJvXunESN1U5RzroJmZW3aZmc2rNEwXjwEkQAIkQAIkQAIkQALhCVQoXU5OPKyhHFGllhQrUjR8JIaSAAmQAAmQAAmQQDYRCNF20AXWmIa5l8zTgiS8HVu0aOGVZ/z48Z7HpAaefvrpcvzxpotwmmEymq+//lp3vfXIkSOlS5cucuWVV8qWLcZbMM3gYbl582a7oKv2hx9+qIe8tXppIgDCYJ06dbxj8KZ0DdeBl6QaPCyvu+463U14XaFCBe/cOXPMBDJple4FhtsoXEKkjBFMMenMyvRCqXtKcL5hts90nS5RTgJHnuEeOrBduorpjt1IpFztA2HOVuBQUw84ZvKWMn+UcyT9pvW2LFJSAoeekP5gEoY0OPQIWfXydLusfPHHuHNYqvhBcu4JbaRmRTOmJy3PEHjz+oFevd/a/qo8k29mNHsInHVsc689THvk83QXqVauspx34plSpmTpdMfyakC/C3t7ZX7q8r65WoxjatQV1EHRwvF3/rim9UVeOT695flcLUdevfj8weM8hicfcVxeLUaeznfrBqfIT498JguGjJcx974leA51b3mgR0pm7pGxfcx47Wn/53Q55ew8zYmZJwESIAESIAESyF4CMelRJgvx/9eevfmOO/U77rhDpkwxXnlpNnDgQMFYjmoY+/Hxxx+Xjh07Ws9BjPF4880320lqGjZsaMXHWbNmyYoVK+wpM2fOlFGjRtlu2Ajo1KmTvPzyy7J06VJ7fMCAAfLNN98IhE7MGDRt2jSZOnWqPYY/TZs2FUyCo4Zu5VWrVvUm3UHX8okTJ9oZuRcvXmy9Kfft26fRE17Xrl3b4wCPy1atWkmjRo2kTZs20rlz5/DpFjHdyvcUlkC9thKcMkSCG5dKoHwYMXHTMgmuMN2tjVnBcXMqKxvgCJ8BI1ZKneYYAFJkwz/mj9nANuLYxZxftaG5zhIRc63gf78ZMTTMS8u+HRJc9qMEDj/LdA8vbBLIXpvRf4RULG3Gv0zATuvXRf7btNaeWdi0tUTtm3vfFIiauHGbP3yJ/PUf+NGSnUDADHug9e56KSd7vpm/7CGANqDtQdd6pUPKVJSZA0ZJEfNMwzOjcZ9zZb+ZeS6vWyGnzIUKGc/7XDJ80Hnr+ifs1b/4ZZxcNzQ+cTS07rL/dyeXMGXrZQuZtq3tPhDbyNTZmp+ClvjpRzeVcGL67r2pH8Eze48Udn/vWL8FrXmxvCRAAiRAAiSQLQTyvCB53HHHSdu2be0kMyAEb8Vvv/1WzjzzTA8YZul+4YUX7CzZ2j36999/FyyuQWDEpDddu3b1gjEWJETEXr16WfERU5dDtMTiN3hDPv98qGcFxou86667BMKp2qZNm+TLL7/UXTtz98KFCzM1Q/fll19uvTeRPxgEViyut6Z3Qd0ImJeuIqYrdV0jSM7+RGTxRJET/V5exptx3ghzBpRF89eMI2nHkrR7qTpj2qa3So2JY3ZqGtUivbVGDC4YK1KpvtkNbYbBFanjggrGnUzA00XTj3Vd2ngnli5xUKzRQ+IVMv+gZ9ZKFC0uh1euaZPBS3F9I0xSkMwsVZ5PAslFoF61w6wYiVxVKVvRfgRZs2V9pjN5lEn3/JPMxxtj67ZukDcnfZbpNPNiAsdUr+tl+2jjKem3q1t3kUoHp/YkGDljvCzgRx8/Iu7ncQJPdLvbK8HaLRtkxIxxMnv5Apn5zzwbntE94p3MDRIgARIgARIgARLIAgKxeEmGKkFZcNHcSAIzbkOEVDHuySefDBEkkad27dpJkyZNrLfk5MmTZd26dV5W4cGIcSIvvfRSadasmReuG+XLl5f33ntPnnvuOZk0aZL1atSZtREH3aWvvvpq2927dOn0XfHgoVi2bFm5//77PU9JnFeqVCm58MILpU+fPva6KpbiWLxWr149GTJkiDz00EO2e7men+F4kkXNBD/7dkugUVcJfj9YZNUc48V4rJ5uunIb4dWMDZkttnuzETe/FznMCI9q29dYwTNQz3QHwgzbhUrokWxbT/pzupQ96OCQ9AsbTw94G6j9vOh32bJzm+56651pngdeQAIbu0wa/T57Rnq06iK/L/tTxs42TGgkQAL5isCPf82Ud74fLqfWbSzDpo+WrBAjAahetcPlrnP/Z1nN/3dxgRUkP5g6UhrWrCfoFj/466Hp2s5VrS4yH3sOt+F/rlxEQTIdIQbkZQKVy1SQww5J/bCJcvR6q59898e0kCJldI+EROYOCZAACZAACZAACeQAgaQVJCH8xWoQ4+BhmJFBWIRYCYP4hy7TNWrUsF2qMzoX3pO33XabXSBGLliwwHo04vxKlSpldLrtOv3jjz/aWbmXLFki1apVs9fWbp4zZsyImMYJJ5xg8xoxQtqB888/X9q3by9IH16Y8NisUqVK9NPMjNlSpIQEjjxLZMkPxlPyYwmUNeNelixrzwtUP9HMmH1ixDTCqd7onQ1L9ZBMnebd9tiGx2TaYt0lU4z3JTw696d6deKcFHhNlqosgYZmzKOAyVsh0608m+2GNx5IdwWM8bbome+88L6fDJHflppZxLPJ3pg0TLDQSIAE8ieBlGCK3PnBgPxZuCQo1YoNq+SKlw70REiCLDELJJBjBA4/xPzflmbbdu2QHxak/5+S94gS4poESIAESIAESCBZCCStIJndgODV6E4EE8/1ihYtKhh/MhHDrNxYssvQxRwCbVxW1HRX3r9HAk17SnDkTZIyd5gUOvkak0TOjgcWXPitGVtysQSamZdKI5JKYeO9uT+ukiRd5INLlJLjatWX/UaMQDfsDds2hc1j+VJlzIRIheyxLTu2yr6U8AUvd1AZOdx4QcAbAi8XS9aulO27Q2eGD3uBDAJrV6ouh5vZOHfu2SV/rlwom00e/Fa8aDHB5DtqG7cbD1dVn9MC0fW9GMYmNQaxedOOLWlHQldIB95MJU139X/WmuEFNvwXcTw9lBljwcL0mrgGvJ3AbdaSP8J6r6JMdavWsWkvXrNc9uzbG5oJs4d0tLs+yo4FHwmOMC93tStXlwXG4wycs8JiYRztOpgptbRpT7B9+/elKzPyXb5U6oeEbbu2pytvyWIlBAsMXrk7du+02+4fpKHc4MH3j+EWzjPYPQfbGBexZqVD5bDKNex1l5g6Xblxdbr24T/Pv48hEMqZOoW5ZaxqvN6Orn6k/GvS/Ou/JQJxL5y5bWXT9i02XvXyVaSBOXeO6bq4evO6dKcdVLyk1DdehpXMPbVo9VJT5hUR03dPRn00rFFPcP7c5X9FbOt6DsbWK2vasjVz32ww908kA88jq9Y2k1wdKsvWrbT3iL/96r2m7RdpFTEfzjC7LixcG7AHzJ9E2yKeZ8fXOdqm/Yd5Tuzeu0eTjHnttlOcFO6ZCI91eKmHa6eYsObgtEmB9pvnpD6r3OfTXnOvbzX3gNuedGxDXPPgkqUsJ/d8hIezKmUrCSYCWW+e3X+vWhL2vgl3HsLAq2jaDMfhyoI4GLIDbQgWLT+x1pn7TEP94PcBbfUEM+MybNrfs2yPAPCFRcoXntFgCnPvRRuQBX/qmGdFHfOMxm8j2tL6rRvDpgo2YATTZ7Qb0X0u6u+DexwfF4ukDf0S7Z5wz3G3Y7kX3fi6jfPqmuEUalSoatv4n/8uitp29H7G+W4+Ue8Y7mHF+v9k0Zpl6Z7r2u7xP4Hajj07vXvErTuNi3h6j+g5/jWeuWj3y811F65aGtMzUdPAvQbPbUzSt8qMl7vgv8URnxVunty2WPHg8nJczaPs7w/Oh8iakSVaV/HkN6M88DgJkAAJkAAJkEB8BAqsIBkfpnweGy8mxUqbt6fqEmhxpwS/7SfBOcMkcOzFOVbw4JIfJbj0Bwkc381MnNPYiJF4ATEekvvTi0g5lqlMXAjCyLPdH5D2jVpZgUuTQpfNB033bL8YNP3RL6y4hngdBl0r6CLuGl7e7ju/p3QyY8XhhV5trxGm0A1r8FdDw4otGi/SuskRjeSRi26VE9NeVjXevBV/2y5fWKthUo6pD33qiVo3v/OwfPzjV3rYvuRONzN64kUCNsR0mxz45avecWx0POEMua9TTyv4ueVYakSXx0e8LMN/MR6yPpv04IcCUQl2Wr+L5JrTLzazhl7gjccHceqXRbPl8hfvsKLQlS06W1aaD5yHsfXgnfb1rEnY9QwzhT57ZaqHLDxUv5v3kzzX/UFP1EHEfzeukRvffFCm/pU2tql3dmwb8TCOliLaAPjDIKw1vLt9SPSmRx4vX975mg0bMPIVeWr0GyHHUS60H9gd7z8u7075wjsOYXfQpfdKm2NODZkBGmw/+elrGTDyZW/yJu8kswHBp9tpHeWujv/z6kiPQ7xBPkbNnKBBGa7xAotJpmDofnzVK3fLG9cNEHdMQAgsSPfVCR+lS89tK2c9fqU8cMHN0rL+yTbek1+9LoNGpfJBAISt+zv3kitbXOCJ0giH6IG0nzBtN9yHAbz0PnbJHXJF8/M98R3njTDjEkYbbgEv6N8/mJpnpFut54EhIXA+DEIcxoHraCZoUWEf4XhRf3bM2/LcmHeNIJEqAg6+rI9c2ORsHPbsyCq17ey6COjxyl3p2nuibRFjL7527WPS/KiTvOcPnj3gmcg4uj/0+8R+VEE+mz54oRGCl2HTGsSnOU+Mts+Z7+f/Ihc+faMesuvLmp0vT152r93G8+L6offb7Suad5YBXe+025P//Fm6PNNL6piPCtMfHW7D3D96z/9hnm+tHr3UPeRtY5KxV//X3042poFoG48Mf956s/s/xmgcd92r7eVyR4drbdAUU5YLfGXBAeT58uadbJyPf/pKbn77Ybutf+KtM/eZNnLGt/LFjLHy8tWPes/tytefbK55l1x0SurzA8+1TkNu0Mt5609veU5wbdjQ7z6VPh+n9i7xIiS4gVmgH7jgJvuhzk0CwheuMXb2FDdYLmnawTyb7rFhbn1rpN5nd5d7Ol5vd8998n8yfeFvesiuMcu0inX+thYS0bcTz73oO1WuO6Or3NT2Sjt8gB5De/ll8Wy5/b3+YYcLeL57P8GkM7BrX+9jJ2HEPe7+juG5f+u7j8m3c6dqsnJZs/PMMyOVjwbitxqzbMN+MgL0eYOvs9vh7hF7wPnT2HxweO3a/ubeqeGFQizu/e6j3n60Ddyfd3S4xoqRGg+Td+HZeJspu/8DxBWm7aM9wt6a/JkZcmKYvHrNYyHPfDzz8L/Ei+PeDyuMZqau4s2vlolrEiABEiABEiCBrCGQ6naUNWkxlbxMwExuI8UOkkDt0yTQ/DYJLp8uwd8/Nv2u92d7qYJLvpfgwjESaHCeBOp3NEqB8eKygmS2XzpbLoBZZyf0fU/OOb619/KuF4KQ9kiXW3U3pjVEmm/ueUM6n9w2XXoQVXq0vFBG3/2GVEjzjospURMJ3g+f9H4unRiJ83FsXJ+3vRckhOGFEaKOWt/zb/Q8exB2p3nx1pcneNY9/c1bGtWKFnjxfvP6gQLRxBUjEQleIK8aweMqM45mNPv8tpfkmtYXeWIk4kIQOcWIcR/e9LR90R1y+X1ePjQtCCpvXT8o3UuwHse6sxHr3rtxSIgYifBDyx8iw259wRPzEBarxcs4WrrwsIU4CoPXlvvCiLB2x7XEylq7Ri1001uruIAAd2yxRrUbyMT7P7DlgzeRayo4jrzj1RDRTuNA0HvmyvvTiZE4Du9U1PcNZ4YXfDSNSGt4W46/792QF1PEhefWYxff7olPkc5/08y4rGIk4vgFpKdNvm886/J05YIX6a3tr7LiLryvXMP99sktz9s26AqGiAOx95krHnCjx7UNAWzSAx9akdGfNkQ6iC5f3P6SN4txXImbyIm2xVrG8xXPghZG2HXvW7Do2+nGDO9Zfz5RDxi3V63pkeYDlGO4jnryYrxNeBm61uTIVJEMYW47duNkdhve12NNmVEnriFfEE9U2HePhdsebmb7VmtqyuK/v8DzrGObaxTB7OCuJVpnmgZYvf6/AR5PDf/85zG6KeCPDxKuwcv2pMMPjCXtz5cbN55tMHi/11Nhn8P4ncMxv7jm1jGe834Lee4dF/rcwzNSxUj8frnCtz8ddz8z9yJE6P4X3xEiRiJt1DWewRPvf18ubnqOe7l0273bdZe3bhiU7ncMz/13bxwcMlZkupMzEYC2iI9a/t8W/K5/YOqmQfXQ+8F/qUuNOIrfA9Sla/BAxP9Dk83z7QjTEyOS4d4fe+/b6Z75eB4+aD4uheOWmbrKbH4jlYPhJEACJEACJEACsROgIBk7q/wfs8hBRgw0ouRR5xhPyTvMhDa/SMp04+G2fW32lH3fLgn+8YURI8dL4JgLjXekES7QVbtISfPfe95tmnjxqFGxqvVA+9/r98ljX7xou0ErxMuMZxUmXojV4B2is8PCE/Di53rL8feeaz3+tAsqRIOHjadjrIZuZB/f/Kwnxgyb/o1cYtK99IXbPI82vAQMuOROr7sc0n55/AeCrmcwdOnq3e5Kuw3xCZ6Land9ODCki9Zp9U6Qa0+/xB6Gt8TjxuMOnkmXvXi7jJ/zg54mj150m1dWL9DZADd4ycBjsc/Hg61npB4++YjjTH66W+/B58e+a71M4D2Krpsw1Mud56Z6K+k57hov4WuNJyW8Ta986U7rCaXdCCG+IG9+ocg937+dKGN/Ou7+JGeSAnhEuuaKkI1rHy3wklFDXiCswiAW4+VcDeXCcRi8i6569R5p9tDF0n/ES163YkyWoF5Ieh48aW5Oq3+ITP0+e1aaPHCBwEvps+kHBI+HLrwloRdodOUrUay48c76RK55rY9t77OXzdfLG1Hw4nQvrt5Bs6EvxfC+haedirmIA29jeF7B0DUdHqPwIrr3o0GeBw/aU3cj9rt2yannhoic8IrEPQ7vW4hsyLMa2ls8BiFd6wF1dIvxSOr6/C3yybSvrYck0oKgcds5V9tknzGCP7wH0V7VUK8IwzJt4QEv68y0xT7n9fRYwivyubHvSPeX75JHh79gh6EIKXOMw3xEbceOqIT7rk3DU7V4dn1Kmtcedib/cUDYDImUtoM6Vx5umwczhN9u6j2c4d6BN+Qz37xtvXRf/vaDEA90nUQo3LluGD4ioDs/DGU5/Zim7mErzEFkguFZ873x7FTLTJ1pGnhe4rooC+5t9R4EN322QSxqe2yokHdmw9M8z1cMWQHPvswahKP3zQcfbS/wVocHNH5z3v9hhJc8ZkOHJ70ahn/APQyDp7ze19gHu0ZmSBS1dk7bQVhIW3FEcI0faR3vvajpwOu03wW97S68y+HRd/GzN1svXvWwx2/Ik8Yb3S2Hnq9rDPOCWeBxj+H5AgFZh0dAfd5ivELVvvltsm3LD3/+nAbZZ5i2+/s/fcoLj7YBz294Z2v3eDwX8UzHvQ7vbAy/4v7++Z9v8K4HNxiGl8DHS3jeIv/a0wL/MzzQ+eaI2cDHyt3mXDxj0DbgqY7ZwtXgeem3ROsqK/Lrzwv3SYAESIAESIAE4ifALtvxM8u/Z+AF2giSsEC99iIHVZbgD09JyuSBpvt2FwnUDH0xzAyI4No/zViVX0pw10YJnHKDyOHmBSRgmmMh46mJ4eFS9mUm+Vw/964PBoZ0iUWXt5kDvrRejPjH/xgz9tx/ZmyljAweaq4HUZ9PBsvvaZProPs3Xu7gjQFrVu/EjJLzjqMrGF4OYN/8PtkKfHoQAuEU06USXkKIA69FvJDD0NX0zvcHyFd3vW4Fvl5tr5D3poywHmt4UYLh5QldJl2DR5F2f/3UdAF+evSb9jC6TP5kZh+eP2ScfdlBeU447BgZ5+u2p2m9PvFjuc9MMKT2tuniNXfQN54nCcROiJzKCF0WIUqogNCoVgM9Nd0aYyV2ebqXJ7iK0XNG/zbJeq3hJQ0sup7aIaRe0yXiBCTK2Eki3Sa8heDVAWta93hBF08YxhXFy5waXhbh7QJBFhbqHXlAxIFIie76qJv9+/fLTaa7KAQAGMQUeCPe2r6H3Ye3mmsQmdXQZl4a/77dhZiGcT1bNWhivZMgeJxiPLUQHq/1NvmBWK4GofPru4dabz+UES/m2mVX4+ga3fTPNy/EKIdrJ9Q5xhP1EA4hXkUjdG9cY16A4dkJu/2cq0x9D7diAO7b24znpNqbkz6Tez56QnftffDWDU9YTyAvMMYN3B8QQGEQitoN7GHGS91i9yfM/VFmLJrjdVO+9LTzrAiJLu1YvHEpTeztZlxQiK9+S7Qtol3BM1sNE4B9+eu3uisf/jjKem/7Paq8CBE2JjlCItqxa219ohI84HAfwyBA4eMLDIJNRs9QdHVXHmCjhjELNVzD3DXGq8OHH32OfDXrOxn16wQZfU/qcwsfYDC2oX/oDTcN3cYHFIyXC3PLgv2zjm2GlbUvzdAG+oxEQKJ1lpragb94dkPYgpishuuMMPUIb3NY++NbW+Fbj7vCHkR3v3exxotn/YwZwgRda2EQ2dE1XdPF8wNjzupHj4Hd7rIfqlB/MDz30BMAhg8xKi63Pa55iNcuxlpEW9RnWKg37YHnnk0owp9E7kUkhWFa4B2oQh26J3849UvvKhi6Al6GpxvhDm2nvxn2AR++wtlCM5YtPpBo92bUweDLdtihShD/WDO2ohruASx6XYRD1IvWvvVcd32x+W1TkRTDYrR/4mozfu2/Ngp+B8fN/kGGmx4KKii750KoRDdvPCNhGNbAHVIDYuycJ76x3t0dGrcWeOXrveWmg/upm/kI8+s/c20w7jtsI20Y6hZjbep4konWVVbl12aKf0iABEiABEiABDJFIO+6oWWq2Dw5IgF4JkKULFbKCJBNpFCnlyVQt60ZU/IzSZnylAT/MyqN8YZK1IIbFkvKzHclaJZA+dpSqMNTNn3rFVm0lBEkU0WtRNNPhvMgirnj8yFP+Acf4ptarC/w8LLAQP1qEFNce3vy53YMNoyNddGzN7mHIm7DE7DV0ad4x5GG39wufe5LHeJhfEsVuiDUfdT7WTv2II5hgokHhj2NzRCDBxnGzMMCzy/X4ME41YiSavXMy34kGz9nasghvFiPmjXRC8OLrf9FxxU34THkenl4J5oNCBzq/anh6OL36bTRumu7rXo7UTYyyzhS0vCgQpuAud0XtdsieOjLuusx6dah650GsbZ6r9NsvdS4qZn3Iq/XH2fG/VLDy75rG7YdaJeYNMZ9UcWYX236X+61zXG/T3FPjXl7pCN+4STcR8+Necc7v8VRJ3vb/o17zXh0fjEScdxxF/Gyq2Kknj9uzhTPqxaeyTp7LSaZUTEM9/izY97SU+wa9fKBI0Co2BISKcKOK8Lh2aFipEaHKAtxAlzhvRfP8AyZaYunH93U6yIOkcQVI5E3iL4QK9SCEttvA7yv9HmIZ6F6CcIzTL3HVVQ6w3jrQdSGucK62471+lm1nrlkXrrnCMJcwbC2mXAoFkN3Z20LbllwruuZ6HaLzkyduXmCYA0vN1eM1OPuM76N8dxUz7hUT84DHx/dfOm58a5Rx/rbhfvkyVGve0w0rRfGvucJcGjjx9Y6ILq5dR3uuQcPSmXsiqnaXnDNKfNDP5Lpdf3rRO/FDo1P935b0Hbd8ZVxDdQBegao4d5SAU/DdD1sWur9rvtY47dNDUOcZLXBK1bt85/HemKkhuF3f35a7wiEKW9s41wVmyEiY9xh1+DlOGXBgQ8lJx+e+vHFjYPtGeZ5rGKkHvtu3rSQCe/csidaV1mVX80j1yRAAiRAAiRAAokTyPvqT+Jl55mRCKgoibX54h1oeZdxwTpdZPanEpxlxEQzAU6gphG0KpuuUmYiHDFfxyOaeREIbvlXguv+lpQVMyW4bZVImVpmJu1bRA47XYIBfFE350OITDEvs8ZLK7+aftVH+UqlzaoaS1nxwn/9Gd1sVHiOwCsELyfwQoRgFuu4WHotdHHTFyF05atoBEpXpEE819sCM6H6DR4QmLAHY0u546w9+sULIV2s/OdhH2OTYSyqQ8tVMTMbl7ddA13vPswUHI9pt3Wcox417vmuFxXKVcJ4YerEIG68SNtTF/xqJj7pbA9rl9pIcTU8KxhrWu4aMzPPXrZAjjceJmAGwQyikIqP8DadtfQP23W9Vf0mViREVz/tughB5YcFM9wkvW14+LQ6uonpFlnVikLw4nGFL52VV0/A5BN4yYaAAc+aXx4bYcbAG2s9c340+cDsqtlh2vURaVcy7Q9CqHZndK+3Zcc2d9fbbmw8cNUgHPjbPo4hXL2Q0P4x83wt5z7AcbcLuKaX6NodV83tlq7pQYg96o6zdDeudWbaogqwuCDqNCvtO+MlqZMVQWSC2KliEjylHvr8WXn7hidtGzzZdNPG7NDwtFVzxxbUsOxcQ4TG81LHtCxlPLViMXR5hpiDMuJ+OsmIMeg6DdEN9zEMbWmaMxlLZurMzRM+9rjCkXsMw3/g4wXuXYyN2dp8pBrz+/fijtuJ35Zw7dFNJ5ZtCM1q8LrTLtgahjWe3dMNJ/yuwI43w04gj7Ap82dYUQrCNMahhEFAhRc27CMzuRq8TeHtjTYE7zwIZEelfUT5zfQq8Iv89sQwfxK9F90y4hmlH47cSyAf8MTHWKLIfz3TC0GFeTdeuG33eVPSDGWR1VbLEdjxmxePNXY+lGIyMwjvftMhAhCOyaZiNXQVt7OGp40lq+PL4vxE6yo78xtruRiPBEiABEiABEgglQAFSbaEyAQwniOEwv27JVDLjH1V40QJ/DfbjPk4wc6ILYvMGg4xBx8qgdKHiBQ33bFM/CC6W+/dISlb10hw8zIbJ2jETStinni1SetU0yu7kBEjTdrWI9IIUOZlz1wocl4K8JFHTHc7CJiYiRVduM9s2MwuQIIXSnTbxdiOOlZiRqiqlKvkRcE/9y9d/Yi3H25Du3G5x/By97ARJTFzsxo8iPyeoXoMa7w8PtD5Jttdyw1P9u1/N63xshiOhXfQ2cgKxk5yIZvwFlIho6kRaKYYgVG9hsYYj8aZ/8yzgiQERXgQTls4ywjAR9o0ZhoPFH87wYtxvwvN+KFmtmycE6vBYw9jhKF7Mzzc4NmGSWKwYAyxyWbMtpdMu4wkgMZ6HX88eNtAWIWoDoEZInE8onxVk1c1iJHhBEk9jnXNStXsbi1nogZXBHfjJrqNrtFqroCuYZlZZ6YtuiJFVpcZ7RgTgMBONd22XUES43HCGxofcdBFEyITBEn1eEP7ymqBNDOMMzoX3bb1HkVZIEhCtNEPPyNmHPCiRFqZqbOM8uIeh5ckJnCCYdIRCJLqbY2wrPCORDpVnd+caO17pRFv1VRMxD5EPDy7MKwBPOitsGu21atzjBl2ZJ/5OAJBUicPghee8s1orFG9JtaJ3oshZdwY+WMMylgm7XmMMsYqSLp5zI7tmmlDISDteO91t+z4kPPKNY9GzWKsv6NREzEHs6KucjK/GZWHx0mABEiABEigIBKgIFkQaz2eMkMwtEtRoxfuNaLkSRI4tLEEUswEKqvmSsqaeSIblhrPx9UiG5fYOEF4uMGLsnwd8ym8hfGkrCeBQ46TIAROI6ilCpHwgjNpQ9HMx16R8aCOFBfefBiPatTMiXY8N7ww6qyo+Mce4yOea7qLdXv+VjsOV6R0NDwgAd20Xhx4uY9mkbphYoxJ1zDhALyH8PLoN3TXwwyq+gI5Y/EcOy7Yms3rrZfdtWZCHB1nzX9ubu8Xd2ZajuRt5M9jVjH2p4t9eIapiACRo6jJn3q8YoIDeGShSyw8sOA5udfct9rl1e9Vhhf2l65+2JtEAmIlRAgIfKhHjDGp42+Gyws8v05/7DLpdmpHOceMDYZ6RpqYnRpjWEJ0QZfRV779MNzpCYWhLIXhvZ1msdaJxleRAvu4t1IyGIKiiP1oAg+uPZqEbbPeThZs7N13YHy/QoZfVlpm2mJ2lvknI5TDKw7PBLRjjNGqXmYY1xbPJYyfef5JZ1pBctCo17xnBNodvBXzio0040/2NxOE4T6FIAkPc7e7qV/4y0ydxcPEFSTbmvsV95Z6WyOdL4xQmhXm1lW47uN6jeJFDnj++ePh2QVBEvcvhqDAhzkYPsph0hTEv79zL+uxjQlLjk4T/RDH/9xDWCRL9F6MuYymvav5y6jhubF2vczjzZf7TMXHIojD0cyNHy1eRscSrSv3+jmZ34zKw+MkQAIkQAIkUBAJUJAsiLWeSJkLm27VWFLMP9MpRsCCF2SaOGlcIiXgvdQb+cpuB9JGE8PLNbaNgIDu2Vgw8Dm6Z8Mr0jsvkUwVrHMmzvtJsOCl9kTT7RldtzHQP7p/wQPuqSv62gk6MqKCseDUVhtB8Lh7ztHdmNfH1Kgr16V1I9eT4CWHF8K7P3xCffLJ3wAAQABJREFUg7w1PPBUjHxh3HvizgiKSChLVgiS7ou8d/FMbuj4dkhGx2fMKMmsYBzpGujGiC68mHAG3kCVy1SwUdGtWPMHTyd0M8cYde4spe5kIjjptLoneGIkPB7PfPxKLw0cRz1HEyQRB+ljVlQsEJXQVRwzyaPrJzx6MYs38uwfGwznJmIQWvWFEs+aFRvMx5A4DHWjXe/v+ejJkBl+oyWzfP2/3mEd59ALyOTGP2uXe/VYxTDMSstMW8zOMkMAwSRCGEvv6Op1jadqO1uv6OqqY8ViuAoIkphEBp6sOnGWvx1nJa/sSEtn0IZQhrFYsbRukDqO72Iz2RO68rqWmTpz08loG2NMwkMPXecx/Aa88HWMPoSHG4M1ozTDHV+46sBvDj5cRTLX02715lAvQ3jU3t3xOnsqni0q6I5NG+cWecWHFHTjxUe7auVT7yN42eIDWKyW6L3o1ln1ClHK6Hhor86mYS1iLasbD/e6PteUnXs82rZbvyN+GS8933wgWvQsO5ZwXTntMSfzm2UFZ0IkQAIkQAIkkI8IHHAzyUeFYlGykQC8hYqYbp3FTPdsu5S23pDwiMREOKlrNyxtu7hZFzXnFTaelo6HXjbmNF8kDY+h687oahcdlwlf9NHlDzOnuuIfXtLgmZaRuS+ZeAFxx4DM6Fwch8g05PL7PK88zHysYzJ2b3mB7TbnpoP4jWrX94LcmUe9wCTeaFH/ZC93yxxRygsMs5FZxmGS9ILgvaJjfEHEhSciDF5lajrRCDwcL2t2vg3GhEOzTLd61zCjuRpmZVVBU8Oira9u3cVrmxh/EoZxIzGDbuenQme3bpUmvkRLL9ZjLRocqA90LdS2F+v5C4wIo3aGEYhitWXGE0vtcCN6gG1W2eLVy72kdPIPL8BswHPt2SsfkNf/97idcRZidKyWmbbolrn5UcY7Pou9N1VYRPl6t+tuiwTxWseb+3buVM8b1RXG4/F4i5VTdscbbiYKUXvIfKBBV3QYxl31W2bqzJ9WRvufmW7bavigpDbceEpnlblDKhx2SA07W7I/bfDQ8SFx7LcloSIthgTBMwzW9dRzPfEMXuFq+tw789jTvEl0MJ5jPB5/id6LriiH4UnQpv2G32gdFgOCvH8SNX/8nNxftu7A8y3aZGHh8uQ+U5sfdaI3uU+4uFkZlmhd5VZ+s7LsTIsESIAESIAE8guB9P8x5ZeSsRzZTwCejvCaRFdsiI2YndtbzD7Ccdx2eQxkf37y4RXQXbj/xXfYBWKEzmSpRXXHn8KkI7EIBnipw4Q4akOMZ6U7UDzC8TKF6w7sdne6NHu0OiA6QhDCDNovjnvfJucXKxGILqja7RX7p9U7ASvPIMBA7FALGAEzN+xoM9FOo7RJJvT68MzqfFJb3fU8t7yACBuZZRwhWS8YE4LA4C0LD1kYvCLVMAmETqKkwhnGmsTEHK4VtR8IUkMwPp/7Eo3tXm2v8KL729bZZvIJbZs66ZJGxnVcUaVYkcSc8SE8uAbR4sYzL/OC1JPOC4hhA0MfaDfvc09oIx1MV3O/Qej99r53vbE3cfwf48mmE3yAO7y1XCaYvf2mKLz813D3v5w5wduFlxq8QF2Dd+Clzc6TTiedJYeZ8SbhIavmdhWFNy/uQdcy0xbhfaYTReFeuOiU9m7SdtbxLk5YvB7KrrCImaVhbjvGsAE6Bqm2Y3jyzlm+ICQfsey4nOL1AIsl/YzifP3bd97kS/oRAef4u2sjLDN1hvPjMVxf7wf9sIDzs6q7NtLCEBLofg9D++zb6cZ07fSWs3t4EwYhvjsrM87DMwXPMJi2FXCCl63aN8ajFoZy6G+ait72QAx/Er0XUT71RoeXKXovuIbngyuqj5szxXtGu/Fya3vkr+O9S1/c9BzrlewFmA14pDY040Oquc8+1IF+RICXfL8LbtZo3hrjfn5+20sZjtnrnRDDRqJ1lVv5jaFIjEICJEACJEACBY5AYm+JBQ4TC0wCuUNghhnIH5MAwJMR4zO+23OwfDpttB3g/9haR8k1ZuxFNXSJdceB0vBwa4iIU/p9bLtRn2y6f3//4Efy5qTP7Oyn9Q89wnahxgQBMAzC//zYd+02BI++nQ540cBLE+LIM2Pelq5mUhTkE918rz+zmydSwqMTnnnq/XLf+T3lsMo1ragAEeyipu29F1FcxJ08xF40jj+ZceCC19mnvZ+T934YYbtQHnFILbnRTLqBWZxh8PLB5BOxWqKMY0kf3Rddg2ei2+0TXoM6/p7G85+D8F8Wp85ii20IPp/e8oIVSMoeVNpOcqGTiOA4usuifnVSiq+MsIfutrAbzrzUcsLs7zB4CHUw45qq/RDnrK163pOX3WtFYkxgAuEVHrg6QQ/a1TNj3tKoMa/hXfz+DyPlihad7DlvXT/IzNI7ys4OXtQIp5gMo4sRAOHJ9PmtL8opD1zgTQT01Og37MzPOBGep7gf4KEFgQTjuPoF7VgzBW4Q58AT48OON2Ioxt38z0yo1P74Vt7Mw0gP4/655narxrnP9XjQelD/aiY30g8WibZFCCzvTflC/temq73k8z36mVmijxV4q9WvdoR0OvmsdOKpm7eMtjHMAD5quMMiuJ6+OB98tZ1hHx9TVEDDfqwGTo3rHG2jX9+mm23Pm7ZvlQ+mjow1iUzFwwcCiFAdTzjDSwf1s8B0NQ5nidZZuLSihWHMWdwT+nxGXPyWYDbsrLT7P31KWprnAp4jENZxz8BrdPe+3XZoic4np374wW/YNa/1SffxBHnBMwz3mdqEeT+GeD/i91LHz9U44Z57eizcOtF7EePvPjbiRevJjHTxMQ8fNtBeIZBecmoHe+/gGGYZv+ejQdhMGhs7e4odixO/3xBzv7p7qLw1aZjASxr/I+CjiPvBys04ZsLua+pXJ7NBr44G5gPfiBnj7ezm8MQ//8Qz7Yzu8KBcZsr/Sxzd6N1ruduJ1lVu5dfNO7dJgARIgARIgARSCVCQZEsggSQmgBfvrs/fIu/fOMT+Mw/vQr+HIbIP74Te7zwSc0mWrF1h4z91eV/bdbBO5RryyEVmoiKffTXrO3l94ideKDzi1CMPL62fTh9tj+3YvVMeMpOXvHrtY3b/7nOvM7PmTvC6AN/z8ZMy7t63rWiFF1GdXVcTRv4xhhlMBSc9llNreFCVN14c8NTxG8SEuz4cGPYl2R9X9xNlrOdHW2O8MggJOhYiXib9Io2Ov6fpuN5oGjbFdNOGwKUzTbc03dOxqLn1gjDUjQqSEPUON6It6hKeize3u9Iueq6uX/428Zm2UScYCxOLayjr4yNeSlg0eXj4c3b2bIzjB08feB9icQ0Cw41vPuiJkTg22ohjw6Z/43kKujPe4zjGA3RnfkVYrIbhFzDxE2behTgc7n7E+KuvTvgoJMm/zXhoELYwDiDskqYd7HL7+/09QTIzbXHI6DfNeKAnWHEF3m1XtepiF81EZsqMNDCjNvIMQ7t2u/ciDDMoP+F4aodrx4iXkY2Y8a2cZ0QRWC0zo/BDF95ihxjIKUES14UA5wqS0bpFZ6bOcK14DN22XUESs4JntaFu0SYHdbvHCl7+Zw2uBy/I69/oa2dUD3d9f9273rSIj+eCjp+LfTwj3bEdERaLJXov4sNGo1oNBMNZwODtjMW1dVs3yEXP3ux5U7rHcnMb7CCCf3DT03YWc3g03tHhWi9L6PaOIT0izZCN35HjzEdSfJzCcwJDnbjDnSCh1Of2y1kiRmrGEq2r3Mqv5ptrEiABEiABEiCBVAKhfbtIhQRIIOkIQGzAZCPwjPR7rcCb5O3vP7fH8QIbj6GrXvOHLpavZ00KmRkbs9tCbOxvBJ9rXr3X67KJMSwxwQQMLxb3fTw4RATDSyw8bWDwLhvY9S67jT8owzmDrkn3IgLPvqvNNe4f9pQXN94xLb0TM7kx1+TxgqdvtF1zNSlMsPG7mXACeYd4F6/Fyzie9F3PH79XGdJxx99Dd2N/29Fr9X7nURkw8hWBqKwG78NRphvxaaZ94Fw1eL2ogc1Dnz8rN7zxgISb9Rjjo10/9H47y7aeE+/6zP5X2i67uJYaXoqvG9rX89rV8HjW6Op50TM32dnrIai56UNwRdk7DLrWimVuumj3ECkfH/myQFhQg3D6hvEmav/E1RoU9xr37xn9L5fXJnwcIlZACMCkHPjggMmgkAfXkPeeb/VL52mXgonDHEu0LUKU7vDktVa4dtsIwm97r788OOxp5yrxb4a0Y2c8QE0J9TFr6R+6azzOUocr8AJi3ECdPms8ud1hCzKaYT3GpGOONm7O1JBnbUbCX6J1FnOG0iKOMh+PdJxFtCd8TMoO+/jHr0wbv8I+S90u9LgWPLyveuVu+3sU6dp4hunzCPnVbuBufB1HEmFu23LjZLSd6L2Ie/Oej56Qy168XeD9i+eoGtodntMXPN3LK4MeS5Y1njNnD7zKTkLm5v3vVUvk/MHXyVezJkbNaj/zUfK8wdfb/x/csX0x9ALE5Ktfu9feg1ETifNgonWFy+RGfuMsHqOTAAmQAAmQQL4nYCZH9r3dRCjy/v37ZcWKFVKxYkUpXTp1zLIIURlcgAiEaz4ahnWkJcX8c64L2ha2scayb98+b9m7d6+0fq5HASKacVHhuYBJNdDV8d+Nq0NesDM+O3IMeNvBS/HPlYvinigkcqrpj6D7GrzI0B1Vve3Sx8qZEHjFYbIQGLqQnZMmKMGLs7bxosKLmI7FmBU5yinGieQV3fEw9llpMzQAZt91XyhjSQ/n16t6mBQya4iGeAmN13D9Gf1HeKdV73WaycdeK3DXM2MY/rtxje2S6UXIog10UcQYiRAZcY1YDfktWay4bSeu0BXr+dHiVTq4gvGArWLrQsdxjBYfXkk1KlaVyua8pUa4cQXTcOcl0hZRxxjSAYIuvM/yomHoC8zEvMOIyEvWroy7nedmmROps1jyi3RnPv6l9RbGJDCdhtwQy2mZioP2iglu4HG/fP2qDNtrpi6WyZPjvRf1chg3Eh7PEPcwZEBW/pboNbJrjWci8o7u5Ru3b4n7MhhnF/+nYPTwv8zvqP5fGHdCcZ6QaF3lVn7jLB6jkwAJkAAJkECeITD5lnekiBkOy10KFy4sWPC+qOsieaZEzCgJkIAlsGH7ZtmQBeMv+XFCYMgJkQHjN2EMumQ2TJyBJastpxgnkm8IavAWTNRwfnbNGgvPPHeMzETzGOk8eGvphDWR4oQLx8t6dhkExYxERffa8GyDB1kkT1g3LrYTaYuo43nGkzgvG7riZ2dbyk42idRZLPnpfXZ3b4ImeGXmhKG9+rvn58R1E7lGvPeiXgMfdRKZgEnPz801nomZuU8gwroTm+VUWRKtq9zKb05x4XVIgARIgARIIFkJUJBM1pphvkiABEiABEiABEggmwhgRuSzj2vpzQgNr+acEiSzqUhMlgRIgARIgARIgARIIA8RoCCZhyqLWSUBEiABEiABEiCBzBJ4z0yUdnajliHJYGb3RIZaCEmEOyRAAiRAAiRAAiRAAiQQIwFOahMjKEYjARIgARIgARIggfxAoGjh0O/RX836Tp4a/UZ+KBrLQAIkQAIkQAIkQAIkkEcIhP5HmkcyzWySAAmQQFYQ+GXRbLn7wydsUmu2rMuKJJlGJghg3E6tDySzz0xyRSMBEsh6Ap//PEbWbdtoJ1qZ/Md0OwN01l+FKZIACZAACZAACZAACZBAZAIUJCOz4RESIIF8TgCzaGOhJQcBTDjy1uTPkiMzzAUJ5GMCw6Z/I1hoJEACJEACJEACJEACJJBbBNhlO7fI87okQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkUAAJUJAsgJXOIpMACZAACZAACZAACZAACZAACZAACZAACZBAbhGgIJlb5HldEiABEiABEiABEiABEiABEiABEiABEiABEiiABChIFsBKZ5FJgARIgARIgARIgARIgARIgARIgARIgARIILcIUJDMLfK8LgmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAkUQAIUJAtgpbPIJEACJEACJEACJEACJEACJEACJEACJEACJJBbBChI5hZ5XpcESIAESIAESIAESIAESIAESIAESIAESIAECiABCpIFsNJZZBIgARIgARIgARIgARIgARIgARIgARIgARLILQIUJHOLPK9LAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAgWQAAXJAljpLDIJkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJ5BYBCpK5RZ7XJQESIAESIAESIAESIAESIAESIAESIAESIIECSICCZAGsdBaZBEiABEiABEiABEiABEiABEiABEiABEiABHKLAAXJ3CLP65IACZAACZAACZAACZAACZAACZAACZAACZBAASRAQbIAVjqLTAIkQAIkQAIkQAIkQAIkQAIkQAIkQAIkQAK5RYCCZG6R53VJgARIgARIgARIgARIgARIgARIgARIgARIoAASoCBZACudRSYBEiABEiABEiABEiABEiABEiABEiABEiCB3CJAQTK3yPO6JEACJEACJEACJEACJEACJEACJEACJEACJFAACVCQLICVziKTAAmQAAmQAAmQAAmQAAmQAAmQAAmQAAmQQG4RoCCZW+R5XRIgARIgARIgARIgARIgARIgARIgARIgARIogAQoSBbASmeRSYAESIAESIAESIAESIAESIAESIAESIAESCC3CFCQzC3yvC4JkAAJkAAJkAAJkAAJkAAJkAAJkAAJkAAJFEACFCQLYKWzyCRAAiRAAiRAAiRAAiRAAiRAAiRAAiRAAiSQWwSK5NaFeV0SSAYC3U7rKDe1vTIZssI8kAAJkAAJkAAJkAAJkAAJkAAJkAAJkEC2Ehg06lUZ+eu32XqNWBKnIBkLJcbJ1wTqVauTr8vHwpEACZAACZAACZAACZAACZAACZAACZBAMhFgl+1kqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDeaFBEiABEiABEiABEiABEiABEiABEiABEiABPI5AQqS+byCWTwSIAESIAESIAESIAESIAESIAESIAESIAESSCYCFCSTqTaYFxIgARIgARIgARIgARIgARIgARIgARIgARLI5wQoSObzCmbxSIAESIAESIAESIAESIAESIAESIAESIAESCCZCFCQTKbaYF5IgARIgARIgARIgARIgARIgARIgARIgARIIJ8ToCCZzyuYxSMBEiABEiABEiABEiABEiABEiABEiABEiCBZCJAQTKZaoN5IQESIAESIAESIAESIAESIAESIAESIAESIIF8ToCCZD6vYBaPBEiABEiABEiABEiABEiABEiABEiABEiABJKJAAXJZKoN5oUESIAESIAESIAESIAESIAESIAESIAESIAE8jkBCpL5vIJZPBIgARIgARIgARIgARIgARIgARIgARIgARJIJgIUJJOpNpgXEiABEiABEiABEiABEiABEiABEiABEiABEsjnBChI5vMKZvFIgARIgARIgARIgARIgARIgARIgARIgARIIJkIUJBMptpgXkiABEiABEiABEiABEiABEiABEiABEiABEggnxOgIJnPK5jFIwESIAESIAESIAESIAESIAESIAESIAESIIFkIkBBMplqg3khARIgARIgARIgARIgARIgARIgARIgARIggXxOgIJkPq9gFo8ESIAESIAESIAESIAESIAESIAESIAESIAEkokABclkqg3mhQRIgARIgARIgARIgARIgARIgARIgARIgATyOQEKkvm8glk8EiABEiABEiABEiABEiABEiABEiABEiABEkgmAhQkk6k2mBcSIAESIAESIAESIAESIAESIAESIAESIAESyOcEKEjm8wpm8UiABEiABEiABEiABEiABEiABEiABEiABEggmQhQkEym2mBeSIAESIAESIAESIAESIAESIAESIAESIAESCCfE6Agmc8rmMUjARIgARIgARIgARIgARIgARIgARIgARIggWQiQEEymWqDeSEBEiABEiABEiABEiABEiABEiABEiABEiCBfE6AgmQ+r2AWjwRIgARIgARIgARIgARIgARIgARIgARIgASSiQAFyWSqDealwBHYvXt3gSszC0wCOU3gq6++kosuukgGDRqU05fm9fIogf379+dKzrt3727b6j///ONdH9tovzhGi43Aiy++aJl9+umnsZ2QD2MtWLDAMvjf//6XD0sXX5FwP7/zzjty/vnny3HHHSfNmjWTm266SYYOHSrbt2+PK7Hhw4dbrs8880xc5zEyCZAACZAACZBAegJF0gcxhARIILsIpKSkyNtvvy1TpkyRP//8U9asWSPly5eXww47TC6//HL7z3IgEMiuyzNdEiiQBFavXi2//vqrvdcKJAAWOmYCwWBQunTpIkuXLpVPPvlEjjjiiJjPzYqIM2fOtALJjh07vOQgmKD9li5d2gvjRnQCEHHB7LTTToseMR8f3bZtm2VQqVKlfFzK2Ip28803y5gxY2xk/I+1c+dOGT16tIwdO1Yuu+yy2BJJi/Xff/9ZrtWqVYvrPEYmARIgARIgARJIT4CCZHomDCGBbCEAUeS2226TadOm2fQPPvhgadiwoSAcL6FY3n//fbuUKFEiW/KQFxL9+uuvZfbs2fZFslWrVnkhy8wjCUh+bLcvv/yybNq0Sbp27Wo/muSXat6yZYvAgw529913S+HChb2i7dmzR+BZBkFw5cqVOS5IehnJwY3crue5c+fKqFGjpEaNGnLFFVfEXPL8eM/FXHhGjJnA999/b8VICJEPPvigtGvXzn6c+uOPPwTCdcmSJWNOixFJgARIgARIgASylgAFyazlydRIICwBeN1cc801gn+A8dL1/PPP225D6g35448/yl133WVFyT59+sjTTz8dNp2CEDh58mT57LPPpGjRokJBsiDUeP4oY35stx9//LEsX75cWrRoka8ESXiOvf7667bh4bnrWvHixQVd/NeuXSsnnXSSeyjfbud2Pf/111+2Pk444YS4BMn8eM/l20aWiwX75Zdf7NUbNWoUMuxB48aNBQuNBEiABEiABEgg9whwDMncY88rFyACeMGFGFm2bFkZMWKE4B9jFSOBAd3K4KWCsJEjR9qv9gUID4tKAiRAAklDoE6dOnLyySeHPKOTJnPMCAmQQFwEli1bZuM3b948rvMYmQRIgARIgARIIPsJ0EMy+xnzCiQgL7zwgqVw6aWXSoUKFcISwUDrxx9/vMyaNUu+/fZbCTcQPbpPYlyzOXPm2JflBg0a2O5H/nHO0AVu3LhxctRRR0mHDh0EniToKv73339L9erV5eyzz5ZTTz01bD4Q+NNPP9nxlf7991+pXbu29ebs2LFjSNdGxIOX0datWwXlwj/9GKMJ3RwfffRROeSQQxDFGsbMRLkgyh500EFSt25d2w0U42eqvfrqq3bstHnz5tmg6dOny1NPPSXlypWTq6++WqPZ9b59++TLL78UeJai+2W9evXsIPXRyhSSgLMTS96c6FE3Me7cDz/8YPlBfD766KPlnHPOCTt2obK76qqrBOVB98PffvtNMM4o6vXKK6+UUqVKhb1erO0g7Mm+QIxlChF8yZIl1isV45liDL1atWr5Yqburlq1yrYntEEMN1CzZk1p2bKltG7dOl38b775xo6V2rZtW9uOMBkA6hfjd8EzBeHwGIYhfOrUqbadYKy8Y489Vrp162bz5Cas3G644Qbb1tDm0K5wXyHNTp06SZEi8f20ZYZnPO12165dAm8dtDl4HtavX1+aNGkS9V50y+5ux1MP6A6Lex/1FM7r77nnnrNtEG0O3Zffeuste6nNmzfbNTyWf/75ZzvEBOrMtY0bN9o2j3aPSbqOOeYYOf300+XII490o9nt7GgPehG0Yzy3MNwDul1DVOzcubN91mgctNnx48fbZ5aGwRsdZW7fvr297xCO5zXSwLi+7nMMx+Dtji7daKszZsyw9zae3bhnIrW7rHzGIA8wbXdgHc7LCx++Fi9ebH9T2rRpk3qS8xf1Fk89x3uPrFu3Tj766CNZuHChvd/xO3LWWWfZ9q7ZwBh+8+fPtwvC8HuDZz7suuuuizhmppY9lt8KpIXnK3jg+QrP18MPP9w+WyI94zL7+4Lfuu+++856FaMN+u2LL76wHx3dusO99sYbb9jfRzzbwAX/B+C3vEyZMpbbhRdeGFEkx72FexQM8buPbskZWax1qnlD++7du7dlifsdvRgGDBjgXSaReyOe54c+688880x7r+J/ANQpxuIOV6f6vMGzAYZnr7YvlAPl0baESaPwW+Yahm1AXeEaeLbh9xx1kJHFyjUz/6clwjrWfGVUPh4nARIgARIggawkEDA/asFYEsQMdStWrJCKFStG/CcxlnQYJ38RCNd8NAzrSAtEF13QtrCNNRa8DOiyd+9eaf1cj2yD1u20jvJc9wezLX0kjBdbCExggQHUIcZFMoh7EGswphHGmHRt0aJFdmZH/FPpGsabfOSRR+wLsYZDQMDYaHjhgQCJsSldgycmuoZfe+21brDdxhhL/vg40LRpU3n22WelcuXK3jnwOMALEES1d99919YfDmo50TUS+dDB5L0TzQZesvAycMopp9hgiIkQuPyGF1m83KmBD4QqCA9+g6By//33RxQG3Pjx5M09L9L2fffdJ+j66LeqVavalyDwc03ZPf7445arv+wQ6vDCBUHWtXjagXteuG3MOo060HtW4+BFE+2jR48eGmTXqIfbb79dVKhyD+IFGOPyFSpUyAvGmKkQOy+++GIriKunikaA2DNs2DCZOHGiFbHxHHANgh1ETHdMVeXWq1cv+wIPkc81cEY+XLEbL/r9+/cXvMi+9tprbnTJLM9Y2y26pV5yySVh2SEc912sY5nFWw89e/a09+Q999wj119/fUj5sYMXbXDECzy6LIcTsBAP+XQFCLys33nnnenaT7FixQT3A+5H17KjPaDtvvTSS4IZb/H74RqERpRZn3O4P5GvcAZhEjPwwiCGYyIZCLkQWNWQPobewJh0foMA+8orr1hhRI8l8owJd20IF+edd57930ufe+hq/vnnn1vhCd71riGfELoh9iBPfhEZcfEBItZ6jvcewfP/jjvusONwuvnCNsoBUQjPiVtvvdU+4/xxsI+PTXh2hrNY7jm0Szw7MP4p7j2M0ewaPoxBkIUnrGtZ8fuC2Zwffvhh+5HmzTffdJO32/jANmnSJOnXr5/XhRgfKDBECX73H3jgAenbt6/g/x/XzjjjDG+oAQ2HUIY2jt8K1yC2nXvuuVY8xKQ2ECtdi6dONW/4XcBvL37rYXjGYuIgWLz3Bs6J9/mhdYrnGcqj10ZaMH+d6vMm9WjoX9xTiK9t6cMPP7T/42gsTFyD54aKmRoOlhj7G/UHvviY41o8XBP9Py0R1vHkyy0Pt0mABEiABPIvgWtf6yMjf/022wo4+ZZ37Ds5/ifRBf+bY8H/gbo+8OaYbVlhwiRQsAnAY1AFH4iD0QwvIxBp/GLk+vXrrZcgBEvMFgnvBHgQ4sUFIg5eSOC14zcIF3hRgSgIsQFiDERK5Acv8Bs2bAg5Bd5nECMhmuIf9N9//90KShjbCx6WDz30UEh83cGLHbqh44UBL/bqVYSXUoiReMmG8IX08LKGPMCzES8MeAGE4Z9z/JMPTyUYZr7E/gcffGD38QdlveWWW6wYCQEMacMbBfnGyytelOCVE4vFk7eM0oNQC7GjSpUqtpx4UYL3D16G4cmGiRrwQhDOIJBAsIaogBd5CA140cMHIIgJriXaDtw0dBvXQvr4gYBHK0QWvCCCO16EH3vssZAXPrzA4QUNYiTqDR648BKDAAyPRqSnL6p6DV1/+umn1osVL+fw+kGZ4Y0CzxaIQHh5hycv2hHyMWTIEJsmvITUi0vT0jVER3ilwWMYgiZEeQh6aKfYjsWygmcs7RZtoIcRd8EOfNFu4V2HOse9gjJAgIjFMlMPsaQPIRz3HRadRfbJJ5+0++54i6h7PHfQfiCm4pmEukN7wH2KZwXqPZxlZXtA+0B7wQeOgQMH2ucihESIPnhxf+KJJ6yXGfIBAQHlAm81eEwiDN57GRnqCGXE/YrnI+5z3Psnnnii9QS86aabrCeVppOVzxhNU9fwBIbB+92dkRthaFsQI8GkdevWCEpnsdZzvPcIhBx8tECeUAeoC/DFbxDuT/we6UcBtBUcw8cPmAo9CHM/fPkzH8s9p+fguYzfTQjpEyZMsGt4ECJ/+EjhWlb+vrjpxrONvOK+wnMC7RSiMz74wZB/sHEN7Q9MwRbPcTwLUWb89sArNJzFW6eaBn4X3nvvPdvDAs9s9zkb772RmecHnpv4/yGjOtX2pfeACsFgmNHHH3zwghiJ3yn874L7DO0WbcdfB8onUa7x/p8WL+tE86Xl4poESIAESIAEspNAkexMnGmTAAmI7coMDng5xBf5ROzee++1XTzhXQIxSA0vKvinGS9oEAL9Aha+PEC4adasmT0F/0yjyya8QvBCBoEQ4iAMQgde6OE9iXTQdRcGMREvPejuCSEF3WPhUeUavAwgRuF6anghhmiKbsfwxMO1YZggA12s4ekGoQbd7pAnFWu1mzK6PPu71EF8gKAFoQTdKvV68ByB2ANBFOWFRx48vSJZvHmLlA7CVZSAVxjqQcsBURHej/gCBPECL7/hvGVQNxBUtSwQgyEUw6sPXVBdS7QduGnoNjjC0AUNL78wiBQQluGFhe6tiAOxBQaBFS9xYA1RXA0veZipFMIx4kN48xu6lKJe4GEDQ/c68IHABY9fCN5oY2rIB4RFeE/ixRUeMX6Dp5or4qGLLs5DfjBmK+4Tf/vxp5EVPLW+o7VbiDFo6xjaAKKBGkR13EvwYIMnKYTrcN1vNT7WmakHN51I2xAYlRu2YRDaNQz78MpCncCTHR5CEPrUwB9iEj4cDB482A5ZAMHataxsD+CB5ypEMNz3MFwfno0QD9FlGAICxC7kA4uWC3Hhge3uIyycQTSAkIqhAdAu9VmOoSwgeOA+hoCOa6Fes/IZEy4/eOaiXuBZDaEK+VCDyApDu8JzKZzFUs84L957BPcrPjLhHseHB/WYRjdkPNeGDh1qP15gH95mMPS8gSGvbjuzgWH+xHLP6Wn6oQqsYPhdw3UxJAqec/gd1LrMqt8XvXaia3Qnxv2jhvsF3Y3xG43fBLQ3GHjqRyDch3g2w/AshIcsfgPxO+K3eOvUPR+iu/s/CI7Fe29k9vkRa52iXWHRZzN6G8TSvnAPo5s22gV+01UchziJ/1/we+n3uAWHRLnG839avKwzky+cSyMBEiABEiCB7CZQKLsvwPRJoKATQLc9mL70uDzwAouXW/8CIU8NXhsQZ2AY08xvEClh6OLmNwg9KkbqMfxTriKT24UWLzrwnMSLroqReg5eAPFShOOaFz2GNbwJVFDTcAhO+KcewpaKkXoML4cQvmAY5yxW0zKi25j/enj5xgsYvO4gQkSzrMyb8oCwpy/K7rUh3uHlHy85EKX8duONN6Yri3alRFc5tXjbAbzD4NHiXzQ9CL6wcPzxYg4PyO7du2t06+GEF2JXONSD2iU0XFqIAzFbxUg9B92ntQ4xXpzflIHbRt04etwNw4s6hCeUHbyjWbw8o6UV7RjyAo81lBX3id/wkgshDfeW67nnj6f7EDcTrQdNI7NrXB/PNYxdC89Wv0GgxEcHjCOowrcbJyvbAwQD5EdFdfc66vUYqV26cTPa1u6huI7/WQ6PdtzHeKahzLCsfMaEyxuEPhUhIcq6poKk/ja4x+LZTuQe0SEmMFYjvP1cU89qHcfPPZZd2xgbUMVIvQZ+E1WodZ+xWfX7otdJdO1+8NE09Hnn5hfPOAiqGC5AxUiNjzXGivZbInXqpgGx1G/x3huZfX7EU6f+vMayr/cTxv9UMVLPw/8Y+hFXw7DODNd4/k+Ll3Vm8uWWj9skQAIkQAIkkF0E6CGZXWSZLgmkEVDvG+2a7ILBCwW8B/2G7ng6tiK6+mI8M3j2YCwsLK7hH06IHRAI4CWoQpMbx7+Nl2WYO/4exEMYvEfg0ec3fdFEV2K/uTOG+49hH8Ir0kd3PngOQXzB2JMwCDaxGjw6YTgnXB4PPfRQ662HPLpjv0VLP7N5Q5dxWLjJQhCOl2F4YaEe4UkYaVw0xFXTiY/c+om3HaBLPrrH+w2eHahLdJVG93aMxYWuzxCR8KKFvOL6mgf/+cgT2i1ejOGdBW8XiA8wtMVYDe0UL3d4oUa9+U29ppB+PIaJofDCi6ESolm8PGO5r8JdDx7MYAbPHO0C7Y8Hj2V0hUT7iNWyqh5ivZ4bT9s8XqTD3fsIQ5nwrIq1TJltD/gQgW7jeMagPeIZgY8hsHieMW453W0duxHdtcMZPP6whLPMPmPCpYkwCCbwONRu2xBKca3/s3ceYFIUWxs+JlBEBLMogjkrqCgGBBMqIMkERvQafq85Y8KcM3oNGABFBRQRREUMKBhQMQuKiAKiggFzQtS/38Iae3p7ZrpnZ5fZ5Ts8y/R0qK56q6p7+utTdeibXMd5uVQZK6aPbLfddq6t8yIBQbRbt27uhQT9Eg/ruGBHlcljMccyJy2smN85fI2tivtLMfmLOybunu3FSUYxJLVi6jScdlx/T9s3quL6katOw3lPuuy5EqwqqVWWa9x54uo8LeuqyFdcXrVOBERABERABIolIEGyWHI6TgQSEvDzKSIWIiz64UMczrDl8HfezIe9I9nH/wBFcGROpHyGB16xwol/eGc+Kv5yGQ/8SY38MCyXYXze/JDJqPeM357rE4HJz3kZ56UXPi7OEzG8neVS5Q3xC8slNvltPBggzhQrEqRtBzz444kWNe+pyEMsnrm0KURK/pgTDMEFrzeG3oaDyZAOohlCJ6IyRlq0X4bulot5wdcL3rnylZZnZftVvvbhBdlp06blym7W+gVdD/5a4fOdlbl/vvjyJi1TXBpJ1tH2CA6CpyTewBhiEx5wYbEpSVr59kHow/z1PN++fluprjE+vegnfRVPVIRf5g7kpYL3jqQPew/k6HFJvxfTR2DPXLRcV/CO5VrNH/0Sjz2mWojzJE+ap6rar9T3l6rKZzhd/9Il6gEa3ie6XEydRtOIfk/bN8rp+hEtC9/LhWtc3tKyror6jsuX1omACIiACIhAsQQkSBZLTseJQEICzGvnbfr06VnzLzK8lD9vDC2MCpLew5J5ocLzSvljwp9ejAmvS7rsH17x8okOsQ6nkfQciAE8fOIh1r59e+c9RNRkXx6GPkbnSAyfJ7rs88d6gmz4dKL78T3MNG57KfPG3KB4Z+Xz5POR0f2caXF5KrTOlzdpO4Bv3DDW8HmYv5BItIjMBKXB85E6Ye4/5gvF+8oLMIjUCD8Ic3j1MkTQe3AwvxkRmMvBCHaAhYX+uHyl5RmXRpJ1tA+sVO2jHOqh1GVKwjHXPgTUIYgH4hwvP/As9wErbrrpJje3bq5j06ynzAzHTipylvIaky+feNPTH3mZFRYk/XDufMcW2lZsH+G6wJyxvERDKGWOy3Hjxln//v3dyy6ChPhpQwrlobq2l/L+Ul159v0QT8+kVmyd5ks/bd/w+S7VNTFf3orZliR/0XSrgmv0HHxPy7q68hWXV60TAREQAREQgSQEJEgmoaR9RKASBJiDiKE/vKnGIw1RJ40xTxuG0MIw26oyzoNgitdTKc7DXEeIkaSLMBA31CtNWRBCEZnwMoUnwV+KtVLmjfk2mbMSdnHGMGbvccGwxWKtKtsB3mwEo+EPry6GWeL5ybD4k046yWUZrzwML8pSiB0usSr4z3tGhl8ExJ2mKnmGz+fnY6V9MFVBXD/w82QmaR/F1IMXW7wHYTh/xSz7MuXzfvTDHpOUqZg8cAyiH9dUmBIwKo2nWNpz8iKAF0a+fRU6vpTXmHznQpD0UdC5DtFvafuFgiPlS9Nvq2wfwRueYdv8MTUDAVF46UF+EZHLyUp5f/H9LY1QWAwL5p/F4qZRyZVeZes0Lt20faNcrh9xZWEdXAnet6C5xuUvLeuqqO+4fGmdCIiACIiACBRLQEFtiiWn40QgBQEmYcd4CPPiVNLDiVbK0FmGK/MwF2c87FXWvFchHnMIJ3GW5jx+rksCPkRFGIQRP+9g9Dx+37g8+Hkhhw4dGj3MfecBMMnw4WLzFndS5tHDEEbijCGUDHFG9PPehnH7FVpXynaAkEMkYLwcPQt/fh7MiQaN+Qn0WZ4yZQofzkPDLYT+S9umQ4cWvRj3sI9oj0cWhjduPislT86Tq91yHubspA3EBXjhWKI3Y0nEu2LqwXs1x0WGZeqEXH0mV5m8dxsRX+P6MR7DbMN88Cr3pcT/MQUC80MydQBDhaOWq136crF/3HUmmg7f/bXHB7yI7nPaaae5FyUDBw50m3y/Snv9i6Zb6Dt126pVKyfO+ik98JQMl7FQGn7fKIti+ghe/ARgiV4PqR8C/2DMIRieb9afv1A+o9v9cdF8R/dL+t3XcWXvL14YZ07KuPlLvcd80nzl2s/PHYn3qZ9GI7xv3Lpi6jScZtyy55a0b5TL9SOuLKzzXIcPHx67S3VxjTt5WtZVUd9x+dI6ERABERABESiWgATJYsnpOBFIQYCo0AhXDFEigIif1yecBMNeveDoI4CynQc5PEswHjgnT57slv1/PPgSsfi8887zq4r6ZM5AAokQfObKK6/MepDige+SSy5xAhaeA0mMN/kY6RE4xZv3lPGRsJlbM2x+uLjfHt7GkEweQvGGYkhx2BCjiArdvXt3500a3hZdLjZv0XT4zjkRGydOnGhEjw0/HONBdtlll7nDjjjiiLjDE68rZTtA4EbIQNAhz+GHZvLv56HjYcYbwW4wAuWExcCxY8fa2Wef7bZRB2Ghwa2sov8uvPDCTERjTkEZWEf7wgPHR6XNdfpS8uQcudotrL2XKX0oGuTl+uuvd4FICHpDPy5kxdSDDyTCdBBhkY56PProo3MKkr5MXgT1eUNkZD5ArmdE/Q6/qGD5zDPPdG2E6OuUq6qMtIk2TTlol77v0RZuvfXWjNBLuwwbLwZ8kK5o2cL7hZcPO+wwN10B1+hHHnkkvMlNc8C8iZTdR/Yu5TUm62QxX/CSxPx1Nm107Vz1XEwf4UUA1z3adXSeYKaFwGBDvXnz58cDNc31wx8Xd6/waaf5LNX9xXvv480/evTorCzgHeoDyGVtKOILvym23nprI2Ae1+DwEGg8lP11OZx0MXUaPj5uOW3fKJfrR1xZWHfQQQcZLxL4nXbbbbdl7cb0A3HB4qqCa9aJ//mSlnV15Ssur1onAiIgAiIgAkkIaMh2EkraRwQqSYCHL+bUOuqoo1zkV4bEIvbwtpuHCYZs+snKCUYQFRePPPJIF0kV0ZKHT4LhMCyPyeH50Uz6rKuMMTcRggniSd++fZ2nGZFymQeM8zKUlwiq/kG70LnIjx+qjkjIfIUM4cNLi4dO5hnD04BgL2FDSCIf/PBnDkTKiUCK4VmBgMJDAt425I90GQ47YcIEl1fmwPRzQIXTDS8Xm7dwGn4Zwenyyy93+bn55psdNx4U8RwjmA/8EHt5kKislbIdnHzyyW5YGg/McG3Tpo0Te8ePH+/mxKR+yLc3eB9zzDFONCeiOH+0WdoFEbnxuuSPIa2Fhkv7NCvzybyXCHjMGYjARL5pS8yZddZZZ2UJHrnOU0qe+dot7RiPQaIhMx3C9ttvb0zlQF9A4IcfD7l+Ts5c+WV9MfXANaNPnz6urqhngh1RV1w7EPNgFuclSZkQp6+99lp3DWAYMNcw7JxzznF5p0yIcJQJe+mll1wboA8QAKkqjSkcDjnkEDc3IcI6UwwgiPlrAVzxLI8KVgyphQMeUPRLPAxhRJT5XIaAyXUZsfWUU06xwYMHu4AynIupKUiT6Ti8d1wprzG58uTX77nnnsZcmghSiGGFvIP9cf4zXz2n7SPMJYvwSNviGkH/ZAgs9xDqgRdKXHvCxjWcFyRcS5hvGKERzvmCJnF8vj4XTj/pcqnuL9wj8UCHw7HHHuvmNOalFf39m2++cXOcct8vhSE6IqBxLtovzLk24p2JqBZnaes0Lo3wurR9g2PL4foRLkN4md8PvGihTzE/K/P20ia5VvMCw/92CR/Dcqm5RtPnezGsqyNfcXnVOhEQAREQARFIQuDfV9RJ9tY+IiACRRPgQZWHWH7o8nDCwxkPxAhCs2fPdg9uF198sZtvMRqQgyGJDzzwgDuWH8v+LT2CEA9R/fr1cw9ARWfunwN5iHr00UfdAzoiH+fkOx6bPPjjmcgb9yRGnu+44w433yAPofyYx4MHrybmwfPDgqNiAQ9ze++9txMtCbBCWcMGP4KtMDcSYgBCDkPW8BzDQ47thazYvOVKF4Fn5MiRLoI25WHYJg+InOeEE06I9VTJlVa+9aVsBwwPpv3xEEv9UM88eCGkIjAxvYD3xiNPtA0fLReRFU8xhh5Sj4hWfvL8aH3mK09ltuGBRd7Jx3333efESNoW7SGJpyHnLiXPfO0WsYq+06tXLyfK0zZoI/QJoq6T5zDrfFyKqQdE8/5BQBGmZeBlAMzwloQX0am9iBY9Lx6OHIN4Qp4RmbwRKfmxxx5zHt+0B9Lhj/bTtWtX1/c5b1Ubggx9jHMhaFE26vWCCy7ICKKI5FEBCM9mPMLJL20/7DmaK8+8SKKMMOHaw3UMD3X6EtckhCFv5KGY658/Ps0nL3oQ57Bi5nfNV89p+wgiMPc5RHiCeL3wwgvuPsJ1gRdwMMG7NmrHH3+8u4bAk/rAu7CQ5etzhY7Ntb0U9xfS5iWVHzrPvJ5cI3lxiLiP6Foq46UfQ8xhy+8I7kO8qOQ6ER0278+Ztk79cfk+0/QN0imX60euMtF+mfsaoZzfWdwrub7wwjbXPOBVwTUuf2lZV1e+4vKqdSIgAiIgAiJQiMAiwRCn+MniIkcyBIoJnvkBz49fmQhAIK75+HV85vrjodj/0bZY5pM/PHX8H3MNtu3Ts8pg99huL+tzaO8qSz9fwnju4MmAGJA2+jLH4mmBl6WfQD/fuYrZRl0whI4+z0NmZYyhjHiBUtYkXmCcC6GAP7xrcnl6+HQRc3xk3bT59GmkyVu+c8ANTz0EEh66qqp+yEMp2wH1g7cc3o35hCT6NO0WwaAq218uxnieITAhQiLm+XZKG6WtVsZKwTNJu6XvIh7gAeaHDqfNd7H1gFcy9Uefib74iMsD12bu/bQN+mLdunXjdnP9Gw892k+xfTE24YQruWfQhnl5ktRDl/sL1zheslAXafoqZWVoMkwKvaQp9TUmDgnzFDPnK4IsQnNaS1rPafsIbZ05RbkWFvJcR9jmRRgez3gQJ7UkfS5pWuH9fL1V9v7iy1TZ61M4b3HLTE3AdYUpK/wLorj9ouvS1mn0+Oj3NH3DH0vfXZDXD5+PuE+ul1wr6Ou8vEtqpeYad95iWFdHvuLyqnUiIAIiIALlReCIvmfZ8NefrrJMPX/iAPd7hN8k/o/f2vzxktZ/SpCssipYOBLmoThqfh2fuf68GOmFSP+5MAmSUW76LgIiUJhAVJAsfIT2EIHaTQBvzf32288FEGIuS5kIiIAIiIAIiIAIiIAI5CNQLoKkhmznqyVtEwEREAEREAEREIEyJUDwEubjw0499dQyzaWyJQIiIAIiIAIiIAIiIAIVCSioTUUmWiMCIiACIiACIiACZUsAr0jmzmReO2yfffapdGCzsi2sMiYCIiACIiACIiACIlArCUiQrJXVqkKJgAiIQO0ksN5667l5F5PMf1g7CahUImBu7lTmTmS+QCK3Ey1cJgIiIAIiIAIiIAIiIAI1iYAEyZpUW8qrCIiACCzkBIhWLROBhZ3ANtts46L/pgnEs7AzU/lFQAREQAREQAREQATKi4DmkCyv+lBuREAEREAEREAERKAgAYmRBRFpBxEQAREQAREQAREQgTImIEGyjCtHWRMBERABERABERABERABERABERABERABERCB2kZAgmRtq1GVRwREQAREQAREQAREQAREQAREQAREQAREQATKmIAEyTKuHGVNBERABERABERABERABERABERABERABERABGobAQmSta1GVR4REAEREAEREAEREAEREAEREAEREAEREAERKGMCirJdxpWjrImACIiACIiACIiACIiACIiACNQMAvP++tNem/qOPTfpFXt16tu2XP2GtuWaG9vhbfe1JZeom1WI6V9/Zlc/ekfWuqXqLGmrL7+K7bjB1tai2UZZ2/jy1Lsv2PAJT1dY71es33gtO373Q/zXrM9Pv/nCrhxxuxEU7eoDelmdxZfI2k6+B4x9OGtd3Jct1twkKM8+mU0n33up/THvj8z36MIpHf5ja63UJLo69vtvf/xuL0yeYGMmjbfJn39s26zT3NpuuI1tudYmtugiFX2pPI9llqpvl+x3ii22aPY+A194xMZPect23mRb69Zy98w5/XGZFZGFfBwju2a+vvvpZLt33CP28ZczrG5Q1y2abmR7Nm9jG6++bmafuIUxE1+2oa8+aWuvvIad3P7wuF3cunCe27doa+2bt43d9+n3XrRHXnvKbeuwxU625+ZtMvuF0/Ar6y5Rx5quuJq1Xr9l3ja3wWpr23HtDvaHxX7GtenwjosG9dPn0N7hVVnL3/3yg507+LqsdXFfqO/Lu5+W2XTLUwNt0syPMt+jC3ttuYvtvlnrrNVJ6+vCoX3sqx/mZB0b9+W8bsfZysuuELdJ6/IQkCCZB442icCCIjB69Gj78ccfrUuXLu5Hw4LKR00578yZM+2VV16x9dZbzzbddNOSZvvRRx+1v/76yzp37lzSdJVYzSagPrpg6u+HH36wp556ylZddVXbbrvtFkwmdNZKE6iJ19WpU6faW2+9ZZtttpmtu27+h8tKA0qYQCnufeVYroTFr9bdJk6caG+++aZ9++23tsoqq9iaa65pG264oS299NKJ8jF+/Hj77LPPbOedd7ZGjRolOkY71TwCf//9tx179/n28GtPZmV+xOtP2x3PDrYRp/W1Jsuvmtn29Y/f2uDxj2W+Zy/8z/Zr1d5u7nmBLbLIIplN738+Nc8xZm023DqnIPnAS49mjt1t0x2sY4udMumyMC0QSHPn599df583N0uQfPCVx+33P+b+u0Nk6ZAduyYSJH+d+5t1uuYoe2v6+5kUnn//Vbvq0b526I7d7JoDz8qs9wthHs0CUe3oXXr4Te7zlUAUpkwIw2FBMnxc1gH/fMnHMW7/G0f1t0sfucVoA9QXn6PfGWfXP9HPburZO+vc0eNvCI596cM3bPFFF7ODduhiKzZYLrqL+x7O86TPPsopSF7/eD8nhnNQsxVXzxIkw2nEnaT7th3txkPPyxJ//TEIw4UEyfxt2lwZ8wmSv/z+W6I2CKOwIEk7eTYQdnMZYm9YkExTXyPfHGPTvpqZK+nM+hP37ClBMkMj+YIEyeSstKcIpLAAjgAAAEAASURBVCbwzTff2D333JP4uMMOO8waNmxoL730kn3xxRfWqVOngoLk+++/b4MGDXLiZYsWLRKfqzbtOHv2bEMg4gdAqQXJ5557zv744w8JkrWpwZSgLOqjJYBYRBI//fST6+uIQqUUJG+99Vb77bff7KSTTsp68CsiizokAYG462q518Gnn37q2l6DBg2qVJBMw6EU977qKleCZlG2uzzyyCP2xBNPuPzh3fPee++55eOPP9422WSTRPl+++237Y033rCtttpKgmQiYjVzp8uG3+rEyHVWbmp9ep5vmwSeceM/esv6jBrgvP6OvvMce/zMuysUDiHq5YsecusRdB4LBJDbn3nAhox/3Fqt08IObt2lwjFbr725/e+wCyqsj3ph+h0QyAa9PNJ/NcTJqCDZPvDme+2SYZl98Pbr8+QAa7VuC7sp5NW2dN16mX3CC3cddblttsYG4VVuedWGK1VYF13x199/2TF3n+fEyC2abWyX9zjdiWkTAq9NPDDx3ERcyyeIXT78Nuu4xc62WqOVo8nn/J6WY1xC7wfioBcjL9j7RNt/2/Y2d948Gxzwpk38313nuXps3KgiBzwKX57ypksW71rE3f/udlDcabLW4d03aeYU2yjiffnxl59mxMisAyJfwuX+/pcfbcTrz9hdzz3o2gjeofk4R5KK/Rpu0+EdwuJ6eL1fxsMw3AYnBl6PPW873XnzvnjBEL9b4Am7WGY5vHDMrgdmieV+W8OlG/hFS1tfw0651eb9OS9zfOsLuxuevHcffYVt2mT9zPrVllsls6yF5AQkSCZnpT1FIDWBecHN6Ouvv8467pdffjH+6tevb0suuWTWtj///DPre5Ivs2bNMv4+//xzW1gFySSctI8IpCXw/fffG55cq622mu20U7YXQZq01EfT0Kr+ffGAxkuMFw9z5861unWzh9RVf44Kn/HDDz+0V1991bbYYgvbaKOKQ/oKp1Bee5SiDmoDk1wcakPZyqvFJc8Nou+oUaOsXr161rNnT9fffv75Z5s+fbqttdZayRPSnrWewJyfv7cbAm84BJcBx1xt6626pivzThu1spUaLG9tLz7AXvv4XZs6e4YbmhsGwjGIbRifW621qX00e7qNenusPf/BK7GCZL1gaLc/JpxWruUXP3zdGLKNEEXazwTDer/84RuXN38MQuPSK/4rNuJViDGMPMm5EB6T7OfPF/5kiPtjbz5ny9ZbxoaefIvVX3J+PtoFw2z7BzzbX3m4XfzwzdZzx70z28LHs/zz779Yrweusnv/e210U87vaTnGJUS9IvhS9mPb/SsmMvz6oVdH2YdffGKj3x3n8h49HpGYYxl+/fhbzzmhuJAgiXcgQ4gfCI69eN+Ts5L0orPfJ2tj6Eu03Js33dB+mfur3TlmiA17bXSlBclwmw6dtuAiQ+7Dbei7QCydb//2kXyJLFd/2azj4/ZNW1+rR4RGXkxhlWnvcflaWNdJkFxYa17lrhYCK6+8sl166aVZ5xo2bJj7ccsQ4B133DFrWzFfEEqaN2+uN+7FwNMxIpCHAC8Oxo0b57xuKyNIqo/mgVwGm/hheckll7ipGWqCGAkyXkDRNldaaaVaIUiWog5qA5NcHGpD2cqgqxeVhWnTpjmxgN9Zm2++uUuDkSz8yUQgTODtf4YZM2+gFyP9duYQPKfLsfZ9MD8ew5KT2AaBlxqC5NzgZVkpDI9IbP9tO9h7n35o/Z5/yHlgVtYTrhR5I403p010Se2z9R4VBMeWgUC74WrrOM82PAO3DTw2o4ag9tmc4AVCwIwhtlHvz+j+pfyORyw24+vP7Z0ZH2R5iT5zzr32R+BdV2exJSqcEiFy8Mvzh+z37na8E4o/CObNfHPapNi5HH0CDLcfGgidD70yyjhuicXmSzp4mQ4J0mNuUOaNvGfcv96u/th8n9utt4UTJGcGwnVttmLrqzYzWZBlkyC5IOnr3CKQkACek8wpueyyy8YOJ8w3HxEeF3h64ZG5xBIVb4ZJssANk7nbOH/U8Cr69ddfjWFshYx8LLXUUlanTp1Cu1Z6O7wQFwqdCz54OyyzzDJFn5PjYVvoXEWf4J8DGVLKG8fqEk3Ssvn9998Nr+BCc2rRXvBEi2tPlWWEiMhDfdT7mHRpw9QzDEttlemj9CGGIsODvBdjtEHaxeKLl+62Dkv6fqH69PllfybKj2uf9H3SSZK/yvYn2hb1wbUmqcW1l/CxtB3yjpdUsZb0mkT67AuvYttDoTymYVyKPsW1gTrJx69QHfj7EPeayvThNO2DdlvZ8xWqi+j2Qhyi+0e/p2ln0WNzfa/ue080H2nqnnsQ+eU3T1L77rvvXD3H9TfaALbGGmskTc7tx/2T63qS30Y+4TT3qEL3HJ8mn9xj6INJmFTFvSScl9q87AVJgoPE2Ul79oxbHbvuh19/cmIhGwnGEmezvv8qawg2+zBcu8tWu1XY/afffrFH33jWCVd7bbGLrR94byJIIlKWUpB8KvC6nBoEdAkb3p5eAAqvjy6/Pf0Dt2qdVZpFN7nvzAHIUNu3pk+KFSQb1Wtgx+x6gBseffagq91cmsssWXiO1zQcYzMWrGy59mYuTwy9bn/Vf2z/Vh2c8Is3KnWSaxg9wXvwWmWYO+Xbe+vdjWHn1EtcQCN//rqL1wkE152dKEkAGx+0ZtwHE+yzb2cbdYxXa1qjjWC56iBNeoij3lszfByeoA2CgDRVZW8HgnD0vGss39gQW70VW1/+eH2WlkDpnlxKmy+lJgIiEBDgh/ADDzxgkydPdj8oecDGszLsrYWXzIMPPmhdu3bNWs8P7Icffthef/11JxLxAMcP6r333tvWX//f+S7iQA8ZMsReeOEFO+6442zs2LH2zjvvuB+ziDnM29atWzc3xPGhhx5yw5b4Ycw21kfndePH7eDBg23SpEnuIZvzrb766rbvvvvaBhvMn2dmwoQJbq5NPBAOP/zwrCwx51L//v2tZcuWdvDBB2dti/tCvh977DGbM2eOe2gl+ADDrJZffvms3fE4efLJJ+3dd9/NCJKtW7e2vfbaK68IwAPG2Wef7YbHM3cUHq/MFcqDTLNmzdy58IzNZWnLyoMY82MSTMF7apA++WQuqrBdeOGFrtw33nhjeLWbhP/88893854x51UhS8LGc9h4442tXbt2dt999xkBFsjvCiusYIccckiFdsYQ0+HDh2emMeDBGy/hjh07ZkQs6u2CCy6IzSvHDxw40Pbcc0/3Rzlo+/QB5l995pln7KOPPjLy5MvJg+mIESNcG/7qq6/cedh+0EEH5RTaGKKHZzNlwQhmcMIJJ8TmqTJ9lPok/wwX5lyIecyNuM8++ziG7uQF/nv66afttddec/3Q93H6CX0Moy6vvvpqJ8717t07S6i94YYb7OOPP7YePXrYttvOf+DhIZr+Q5AoeGEIpe3bt7e2bdu67/yH6HHOOec471Ha4dChQ93+9APOTX3QTrlG0Ie5HpE/5ng98MADM95Fvh0x3UQx/SmToWCB69Tzzz+fuV7SDrkuRvtJ+Bi/TJ/moR1W3hA2mDfu2WefddNssJ6XP7Ao5N1O8It+/fq56zUC7eOPP+6uEzDIdU1C+KOtMi8ww/wRQJs2bWoHHHBApj5ffPFFdz3lmovRn0aOHJnVJ9yG0H9pGZeyT9H+BgwYYDNmzHAeqHh00t7iLK4O2G/KlCmuDj744ANXD/DcfvvtHVuuIUmZJG0f9EXqgbbE/YsXTbShfNd1Xx7ul7R57m/cT7zRly6++GI3xJ77UdjOO+881/Yuu+wydx8Jc0haNtJLeu8LnzvfMhyS3Hvuvvtud38666yzXLApnybz7DK/9f777+/qy6//8ssvnUcy/f2oo47yq2M/C9V9+CDukdwLOAbxDQ9Gfo9E7+m+fTNfLNdOrnXUM9df9ie/vGAkAM2VV17phHTOQ71yv+daxfWNNsL1l/pk6gRv9ON7773X/bbgmoIIuPvuu/vNFT6T3qMuv/xydw3hOk6fyve70J+E9sN9k/si1zN/LW/Tpk0FUb/QvcSnqc/cBKZ99Znb2DAQxtLan8G99/j+F7rDZgfDqF8MhKq5QdRqvP46BeJSnOFJ54/x25dfplGsIDn89aecZybDnxsFc+khlDEMlaHEr3/yXhAFfBOfRKU+r3+84vyYV/Y4M5Eg+eGsT9y5V8oR0GXFZZZz26fMmpYzj3sH3pV4HBKh+7IgwMzl3U/Pua/fkITjN8G8np9Egpps0mS9jNCIh+LgE/rYWYOuCeaAfMJ5JuKdyPyOR+60nx24fecKfY7ze69VhEiMoDsIkgRFYig20a/jDC/bA7bv5ATJQS8FvwH+iaI96OX5XrAH7dDZzUMad6xf9+mcL+x/owe6r9//+mMwhP8l593JCoTdylq4TYfTanHBxlUqSI4MRFX+wtZpy12zBMli6yucppZLR0CCZOlYKiURKDmBK664wv2o3mWXXdyPY0Q9fuAzpx0RpTEeTL1nms8AP4L79OnjjkHgWGeddeyTTz5xP5BZf+qpp+ad+8i/Tb/55pvdAwZRIRFpEO8Q8RDgmKCdBzTEKEQNHgII4EP0ST+vEm/7KQNihI9Myg9j0rnpppuMBxjECx+Ihod4vFfCnoaIGZTPD5XyZYz75AGIvJPelltu6YQQ5t5C0KTM3oiSiWjHgwA/zHkQ4Yc7ogFzfv7nP//xu1b45CGN/JBX/nxwDc7DAwIPDTwwLLfc/B9O0QTSlpUHIJgjTBDkCD48QN1xxx0u8BEPW97IF2WKGiIT2zi2kCVl4zlQ7wh21DlCIW0Uoe22225zop73iCLPPLgi6NCeERUQZ3ngRXw59thjXdZ8unF59W3dizEcwEMWZaOOKSdiO9FPMfYjHwgZrVq1cgIBD2e0KQQShM84r2E802jXiG4E30DY2mabbSqI2pyj2D5K0Krrr7/eCZGkTdRo+gVtirwh9hXyTGR+S8QoyszLANofdUGeevXq5fpW48aNnZBI2+YBer/99iPbrg0hfPFSADbe6JfUIdcMBBUEYl5q8GIEsdELcb6eyDP7+z7n8891husD1x0e4BGOEAwQhagXBF7Mp1Nsf/L5piwEA6GuEAzJt+8neOSGxSF/TPiTNsS1I2z333+/uy7Q9+jniM+0HcR39t9tt93Cu2ct+z5HRHDODx9EC46PuybRjm+55RZ3HYUX+eVaxHWJa8oxxxzjBFuul7RN+hjsETepq7XXXjvr/OEvaRmXqk/RrxF0uCZRJvJO2/7f//7n2kD4Ok9+4+qAgCu0Sa7RiJC8lKMPIxLzAuTkk0926RZikqZ9UL8IW7zE4py0V9otUZILGf2JctAPw22O+yXr6QOkR3kwxDn+uF9577wwh6T1nfTeVyj/4e1J7z38FqGvcZ3lOuaNPk1ZaPNw9MZ1ivX+N4xfH/1MUveeGfVz++23O29Efq9wvYE51z1e+NBOeBmA+fbdt29f933rrbfO1A31jmjHfZWXrLQrjucaR91yb+GaGk6H+vRG2rzUQIhnX15+8XuJFxtx92aOTXqPghkvFxCuETl33XVX1wfIW/R3Ifnh2sFvMuqEezNCKfXEtZxrGS+3vSW5l/h99ZmbgPdII9hFWovzJiPK8/3HXR8Mv40XpTZsvLYdv8ehWadaMoeAdf+L84Wqbi3buf3pD12D5ZuevMeJYqUSJE/reEQQTTvbm5gANUms0dLzR2H9/Puvsbv/8s/6Ff4RJmN3ClZefWAv2+GC/e3u54KXQ0GU8kKWhOOTQbTsE++5OCupcecPNobVe6P+bzjk3GAI9XF27wuPOGGUoDME5Jnw8Xt23cFnZ0Wu/vG3n93Qcuqiy1bz64W5E+H1RjB8/fFgPknqKM5+CQTJ1utv5SK2P/XuC4ZgWieo+5FvjDEC57TdaJtgOPf8QFxxx7OOuUwvGJrtwEBAmbM7H+MCA+U6Lul65oK8qecFFXZfteGKFdaVcgXi4x6b75iVZJPIHJBsTFtfWQnqS0kJSJAsKU4lJgKlJcBD7KGH/vtjgx+1RHvkR2W+H/P84OftPsd7oYec8QYc7wAe5rxomC/H/ADnQdj/kOcBGJESIYkHDLzgvPkftOTNp80DHD/G8Q5AMMEQo/DAQhzBo6N79+5OnMI7kmM5Bw9nGD/WiWaJMMMP+0KGoHDGGWc4wZZ9u3TpYqeccooTABBF/ZxPPBDznX39gzxeegg5eEwgNBQanoXgcOKJJzpBwOeLH/88BCAg5vICQohLWlbyQlp4klAPfrgrDK+66ir3sIXnTvgh0Oel2M+0bBDtaKPeM5aHHDzvEACoS8Q2jDaHUQ4EHgzhCG8gHiYRvJN4ILkDY/5DAD7zzDOzPADx2iMftAMeyDDaIg9vY8aMcR5QPNRFjYc92gOiIYIk+eJ7nBXbR19++WX3gIrHMg+92B577OHEGnjwEB8WNKLnRthAjCSgCeIefZR0EAF4wKV//fe//3WHdejQwQkhlJl+Cyu8fWiLeFP6/o3IxcMt/ff00//1KqCNXXPNNY6ZFyR9fuijeBr5Po9HIg/MBHzgoRevMERCjLLSHxFraDc88Hsrtj9xPOIjZUasQshFjMAQFWhfsKAdRgUwt1OO/3j4p47I42mnnZY5docddnDlQ2jMJ0j6ZBEjk1yT8CJH3Kf/0C+84blKX0dEwAOavsMf7ZL+Rf379uOPyfWZlnFl+xT3BEQYRHCuWd64L/ACoZDRtrjfkG/q1V8faM8I3rQjxHK85/MxSdM+6PPcl1ZccUUn6vshrrwMog8gkuUzPED54x6MEOz7Fv2V6zftAUF6ww03dMmwH8Y1Ps6S1nfSe1/cOeLWpbn3+LxzrfWjN+g/fKfMCJW0A98vfZm5duayNHWP9ykv6BASecnJJ0a/uOuuu9zvFe7L9N2w8bKMvunzxXb6GoI31w6GWnPd56UZ10XK6csXTie8jDCMGIkAj/enF0z5PUb7QRQMW9p7FG2KtlPodyFiPaMJELQpI9d6jOsLjPgtyctYfhOluZeE867ligQ2bzp/1M+s77+uuDFY8/WPc1zk5Yb1lrF6dbOnFCEi8fiLh7rjnnx7nJ0z5FrDcw8vs1yGeLTvNvN/2+Tah/XhqMtEzWbeQYyANhgBTC7d79Sc3nhup4T/7bzRtm74csLds3ZrHsy9OSEIDsM8kHH2xXdfutVxUbzD+zddYTVDGL1k2P/slEAM3HC1tcObKywn4Yin6jld5v+m8gnk8uQkENCJe/R0f4++8Ywdfnsvu+/F4dZ5y11sp4239Yfb8AnzvVYZzn3qwMsy678K2gmG92QuQRIPSe4vPbbby656tK8LnFOvzlIu8vP+23Z0wmehuUrx8Dxzr6PdueoGc042WaGxwc7PR+k2VOK/RRdZNFH7rMQpYg/dNChXkn7hD05aX35/fZaewKKlT1IpioAIlIoAQ4fC5n/ARyN3h/cJL/OjPmz8AEU8iBNhwvv55eiwHn4I+x/viCdhw4MIw9vDG+IFP8LDb+LZ5oc38fDnzQtXPGB648GNBzj29x4lflvcJw8UeI9642HIR6D1w09hx0MlD3pejGR/0icP/ODHs6KQcR68k8LmvUB4mMtnScvKgwKGmEtZvPGQzAMTD+oIE6WyYtg0adIkI0b6fPi2ENdOw20S70Q8V2iT/sHfp5H2E9HRt01/LKIe54iKaF489Xz9/sV8lrqPMpwZHoWmVUB4xPAG8qIH32GPgI9YRfvAaDs9e/Z0ywhbvNTAexnx0IuFbKRN01+pk7DhgcfDOR5v9I+wIdx7MdKv52EcQ8gMp48gyEsOzPdH9yX4rzL9CeEDwYP+F24DCBP0f8qKp2Ya8+Xk03PkeK4biLUM2fT75Es3yTWJ42mL5DcqLnI+REn6EkJ1ZSwt48r0KQQyruWUqW1oqD/5xyO3kPcv+yHs8OKI/b0YyXqMFwyILYiNhSxN+0AspV65R4avSbSrqKCV67zcpxGe8AbFEN8ZGYCATV8MX3cQ5+i/XtTLlWah9UnbWaF0/HafxyT3HkQt6oL7te8rCHi0AeoJr0HEY2/sx0u06DQqfjufaeqe+sWjnpdNXowkDcRApr/gk+k8osbvkvD1gusYdc5L1CR9O5oe33mJgSFgezGS7/Q9f+3ju7di7lFJ7jmIwNQFv9O8GMk5WeaFB/nheo6luZe4A/RfTgLeE/CNYAh0VAz6Lghms8XZnW3zXh1sQrA9alwHEIP4+08wxJf5BGcHwuZtT98f3TX1dz8smAOJtM2cg/wRfAVjvsrH3hrjlhfkf37OxJFvzn+BHc4LEcxfnTr/Hrh5MN9iITu23cEuCM7EwEORyN2VNYISnbTnYVl/PgI5aSMotgmiqOOVGTbmcvQBeF79OPsefv8/QYbwqPV1widzSmLPv/+qff7tv89U4XTn/TX/+a57ID7SdqjjB4Lh2iwfEIiUmN8nfFx4eYX6jZwnId6ECKXM81kqMTJ8nnJcLqa+yrEctSVP/z7h1pYSqRwiUIsIREU4Pzl63HDWcLERMxhWxEMBw4cQCBjyjNeH98wI7590mYcpPIZ4+A8LDRzv88ZDiDd+7POHAEheGA7Mg5rPf1icIl+kgZDCdsQL/1DEkKokFuXFMd4Ly5/TCxN4R+AlEDY8urA4IS28X65lhmhRXuajwkMk/CAQPiZpWckr+eThIWpeDPUPFdHtxXwvhk0cc98WPHPygmcJXlEMv6Q+8XhFLKKd+iFwxeTZHxPNB3VJe6PNM69b1BAqefCsrEXPG1f2uHMgSjM8EO89+gciBmIiD/fekzfuOL+OuuKBl2PxhgkbeaANItb4foqwhdg1atQo54lH+2kbEYpgwh/s8CiCD2IKD+e0Zy82hM8Vt+ynK/DnDu+TlA/HJO1Pvt3y8iLap72wkLau6ccIgXBgPlG88BCNyBMCbVKLtg+Oi16TyBuiKZ7T4RcP/hx4w9NWStnXfdr5GEfznqZPcQ2FPYJ1NB1/7kKfeOxicd7qtF/mXkxiadqHF8rhUqzRlxHAEIXod9zTYIGwiqccL7wYGcCDI+Ic5fP9othzxjGOtrM0acMszb2HMtP38IyGHfduyoQgBgvKzMgHPOFpR/6lXK48pal7pozB/D0xnCZThHAd4kUp16+wSBhlRn0gSDIsmt8mcX0xnHbcMu2H+36Se1qa/hQ+VzTfvu2E77f5+CG2h19Kp72XhPOi5WwC667SzA3hxbPxihG324X7nOh2oP/f+ewQJ1ISZGWbYP7GfMZQ116d/s+OvONsN6T60B27WVj8yndsdJuLujz+Mbf6ziMvD4Snf6doYSVzCF73+F32QDCkm/kLF6TtHOQNb0VERIaSH7/7IS47RKg+e9A1zvuPwCRNll+1YDbxOL3uoLOtw1VHVBCHCx5cxA4MU2fOyD6j+gfBbNrb0nXruVQYZj45aA/Yisss7z75j+HSrwUCKwLgKxc/bMvWWyazjYVDbjnNicdDgrpDCM1lsNhxg5ZOvGQf+DDsW5afQNr6yp+atlaWgATJyhJcAMc/9vjoYCjM5ODhaEPbY/ddK+Tg62/mWP/+9wVeQdvb1i23qLC9Klf0H3C/E2T2369rqtPMm/enDR/xmE2a+IGtvc6a1rHDgr0ppsp8Ge6MqMAccgzpxEuFYdr84c3G23HvoVjVWedHPUOpeAPPww0/0vGMCYuWPg88KPDQz8MLw7QRUXmIQZyJe9Dwx6X99OINXhX8xRniQDFGGXi45CGThw2G7sVZkrIiACFU5BqO7R9CkngIxeUhbl1VsuFhnDwzbQDiCp6diD60RQJA0D5Kab4sPCDiERhnxdZzXFpp1yEyM3SO6QvwfKOf0i7wosGrJ06E9ueg//CAzUMOQU1yGeULi4IIkgw/9J4zPIBHDVGB6waG1xOiBg/AnKu6LWl/QuTBaFe5jP6Y1piSgv7MMEzqiT8EDobScx2N45f2HOyPNxgW9u5yK/75z/f1tKJqOI1cy0kZc3yaPuV5+7znOn++9YhXWGXS4Pg07aMU+eZ+hSjF/QWvPcQ5PDyZX5n7Gv0d70muf5wv39QM5L+6rZh7jxckwyIs13ZEPV60UGZ+DyQZrk1509R9kv7D9ZLrYZKXPcXy5rrKCxxegiWxNP0pSXrhffxIFS9Kh7eFl4u9l4TT0PK/BLgn3H/cDbbHFYfZLU8NtOfef8XWXaWpjZ/ylvN2ZM/rDz430dDozsE8eNc/0c+Yg/C6IFDMJfud8u+J/ll6a/r71vnaoyusRxi95sCz3HrvZYfgtWfzNsF8lEtk7d992w5OkBz7wWsuOvNqjVbO2p72yxkPXBkbsOS8rscZ0bbzGXNDPnD8DbbX1UfZRQ/fFMzBONLNR4knJ5Gj11qpifX/v6vzJZG1jfP1bNOtgtdi1k7BlyQco8dEvyNCDpsw2oi0vt4puzqvyGWWqm8vBFGv8Y5lXsdOW+6cOcwHn2mz4TaxAus+2+zhBEk8H/MJkiRIcBvq2S+7hSr6LxcrRPOooI2HZlz7JGtDTrgpUT8ophjM30lQo6gx/L3njnu71WnrK5qWvpeWgATJ0vKsltQ+/3yWfTB5in04ZWrgZbShNVk923tqbiBksJ1taQ2x8/mxgUfIxefGBnsolN7nn39RlLDw0dSPbey4l2yNJqvb2msV75lQKH8L03YeiJizjT+8qPDSYA45Jn4nwnB1PAQxhxhiJA9lDGHyHgd4riGYRg2vCQRJ5qhkOBf78Sa/VA/+nM8Pw2PYk48sHM0HD+nFmp8jqtCDQKGyUn+IdHhrxJkX03KJnnHHFFpX1WzwiOQPTzaG8DG0HbGHufOIAo6QXirzZcErJ0l09lKdN0061B1Df3lQR8RGuMAjj0AgDAvmBUKcwQkPYgQN5hXMZbShsOGNyUMz/Yll6iLc1hFQEG/xsD766KOzhtUyh1+xnsPhPKRdTtKffF2Tx1xiQLFtq23gRcofwj8vSvC2hR3eivmCX6UpJyInhldrnHmRLDpsOW7fYtYlYUy6nnOSPuXFVZ92Mfny4lGua2DSNH2+k7QPn2/qwtdL0vP4/bjP8WKBUQGkwydTK2CMVKDP0dd9+oh55WTF3HvwiuR+xTWEqVAoNx6/GMIkQ+G5znOd47oVni4lruxp6h6OeGbSTuLuu9wrOaev27jzlWId9cr5c/Xj6Dl8u0zSn6LHFvoOP88k38u+Yu8lhc6/MG/HY23IiTfZzaPvcSIRgiJz6W3aZH07r+uxWXMI5uPEfZrgIgf97xTr9/xDdvQuPSoIVwhdL334RoVkwsPF/XDtji12riBGcuCagcjHUOk3p01yAuAp7f9TIb00K9779MPY3clrEoPToECUvDsoMyLb5CAK+BrB3IYIXscFw7CJEJ7Gzg2E0Mffet5mffdVzsOScMx58D8bEFMfO/1Ou3jYzUaQGQReXuTiqUmQFeaf9MF48FolEjjW7Z/o2v8kk/lgqPeZD1zl5v985aO3bJt15l9PMzuEFjo038l5WPL7LldE9tDulVrMxWrXTbavkC7lj2uf7AiDqrIZX39u/EXNT6nA+jT1FU1H30tPQIJk6ZlWW4p4rdx77yA7q9cpJRNsEArmzPnWqtsX5otAZMW6798t+JHe0M05VG0ga+GJeCjgIRbPBH6MImzwx1DDa6+91oke1SFI8iCG4U3kxUi+5/K2YngbHoE8+PuHNbwmS2l4qWB4JxAEo5TGUHY8E2BeKO0kZUWAYDgVQlDY0408E/kTCw8t9AJUOICP2ynhf1XFBq8RBEiG8jL8FDbML8gf7ZGHVP4Yxs2DI0YZKmO+LIjxpFlKUbsy+fLHEsCJazgM+GQYP3880CLkI9bmEiRJg/LxwEmb833Fpx33iWfS888/7/hzHSD6LMO36Zve/DxvvARIMsefP66qPpP2J1jggYWnVNzw3mLyh1cdbadZs2aONW2XuUi5HuHZikcr83369lrMOfwxCNO0z2nTplUYVso+fkhquK/7Yyv7mZQx50nTp7heUSY8AbneF9P//MsW7+EYLivXB99H4ubmC++bpn14QZt84yFcrDG8Hw9/+jJeaF6co19xDUSQZMQA/Z17QblZ2nsPYhzXb15AEhSGfuHrhRcf3JsoM9chvnPNy2dp6t4L9fST6IgC2gn3T7xWi2mD+fIYt432Qz7i7tnR/dP0p+ixhb77dkzf8Sz9MfymYE5Tfh/CjnykuZf4dPSZmwDzDd56+MXu2jdzzixbqcHyOb3BiG791e3x847vvlnr2G0n7H6o8ZfE+h5xqfGXz0afNSDn5mPbHWT8FbKZN79YaJfE21ut28L4Q7T68vtvbJWG+b2O8/FgiPy7Vz4ee+58x8UeUGBl3SDKNZ6s/DEvJ3+rNlzJGIIfNgTqd3Lkye+HR+tn/3vJf818xuWZ8350/bOZffzCvf+91i9mfcalkbVDzJc0x+Rr0zFJF1zVPAgolKuPhA8efEKf8NeCy0nrKy6h6X3Gxq3WuiIJZPeQIhPRYQuGAMOaP/5kuj33/AuJMvDLL7/aaxPeDIZGP+68IL/6KjsK3EcffRwEGvjGpTVx4vvBkJkv86bLMOspgZcmno3TZ3yaU2QikZmffW4vvDg+mPD71eCH2vxz+MQ/C7wqv/pn3RezZgd5yM6X30+fyQkg7vTr188Nzwwf5UUy3qJVh/kHAIZxeePh9OGHH3Zfmew+angOMmQMb05+SCMIlNLw/kJcYPJ5BICw4Ulx3333uWFX4fVxywwp80PF/HYi7zKPEyJTEitUVh/RE15hERc+CEo8BIaDIfiHEATpsPkI1+F1cculYhNNG28+In4SwTvqPRJtkzyw8zCLsBseospDfb4hudFz8gCMZw7p4HUbNto/UxjwEJbPvCgXzXO+Y5Juw9OO6QyideV5hOs7Lk3v3Ut05mh/Jm0fXIFjaS9EgEcEQERDhKStMAQ53IZpT1i4v/Kd4fXeOzKuz7JPZa0y/Yl+RN4Rf6J1hThA/8ELNY0honMNhVvYYEjbijIP75N2mf5BoCUYIxqHDS54zTJsOSxcee+qNB6IlWFMntL0KTyv8PyiD5P/sCFYRespvN0vI9AjtuNFjbd82IYNG+YixfMS1VsuJmnaB/ty32JqA8Rab/ShaDn8trhP7/VIfVJ3YTGZYdv0O0YtIOL5+2RcOn5drrL57aX+THvv4fyUmesDnLgv+ZeQtAXKST1yj/Vs8uU5Td3zcpVz8YIl2q68V3iSc+bLT9JtTE2CcS0KX8MJ4Be9rqbpT0nP7/fj/uDbcfiazXXw7rvvdtdE7zmZ5l7i09dnMgLUAR6TCB+y9AQQ7gqJkelTrZ4jGgTDtVdfbpUKYmT1nF1nSUtA9ZWWWGn3l4dkaXlWa2p77L6Lvfra6/bwsJG2RYvNg+Eoud3YES5vu/3u4Ef9d4Gn1fLOC5Ib5T57d7Jdd2nr8j34wWGBQDPDLd96293Wfs921qXzv94z4cLxsHbjTbe7/esEPzb/CH6Errvu2jb3j7lWL/jnjR9CgwY/HIiWLwY/GBmO+bf9MfePYPhSa+vcqYPbbViQ/ykfzff2GjhwiDXffJMgqvD8CML3PfCgT0qfKQjwA/OFF14wBDKGFuKRwIMhQ6ExHw06RZJF7crD3bRA9BswYIDzYELgwfvRB2fw8xyFEyfgCfPi8cM5qbAXPr7QMg//BxxwgN1yyy0uojCRxJmvD08CvJ4QFOETN/QrnDYPwtdff70b9o2XAZ4feL0hpkUjkIePCy8XKivbEZbIF8PRENjIH+t4oEVcahYSbKl3vHKIpIwHJcO28ISIPgiF8xBeLhWbcJos8+AFZ+aPZDgy3kIMyUc4J7+IY+F5QikHgvRVV13lhBrESLz3eKhLYwzL5xyIj3iF4LHDAzHnxAMKb5qwyBNNGyEBL0Xa8NChQ51Hk394i+6b9jteiOTpzjvvdDwQLBBIEWt4gC/kGdy2bVvnIYZHEpzYn2s6dU35yCcPx6xDkEPsQoj0Xjn0gRtvvNH69+/vpk9A0ENAoo4QIJl708+HylBLtiPC0WfzzW+ZloPfvzL9CU82H7Dnoosucm2NukOMhCdlZrsXdfw5833ixUbfoh8ReRxvIgQG2OJ1xbWpFN6RPg9EtKaNDxo0yLVNvFi5ZtMPYH/ccce5Yfp+fwQb1iP+0G+JEOw90vw+0c/KMPZppelTHTt2dKIbLyPgSLuhL9FmaZeFjH7AfKr0kcsuu8wFxkJE4Xiut9RRWGjKxSRN+8Brkb6DeHb55Ze7AFxcf/Dui7tf5SoDQqq/rzBMO1xern/UMyJnOP+50mJ9rrLlO6Yy29LeezgXvzMoJ9cJ7xHq88C9i4jSbA+/RPPbo59p6p77B9c2xMcrrrjCRUPnxQ59ld8bXBu53laH8duBF4AMUec6gRDLtZeyU99RS9Ofosfm+06b557LtZz7A7/F+O3A9ZB2zG8U/xunbYp7Sb5zapsIiIAIiIAIFENAgmQx1MrkGDw1Djm4u11x5Q2B6DfUjj7qsNic8WMaMXLJunXt6isvCkSKZQPh8I/gLelAGzxkmK3ZrGkwn8+ads5Zp9pDQ4fbqCefsZtvutoQGnNZvwEPBILILPu/ow+zFs03C95K/2J33nVPIOrMsobLNswc9sSop50HZae92gdzFm7r8jBmzFh76OER7uFw5512tOOOPdKefmZM4Ln5hJ3f+8xAvKjj3rLjpTl58kdmy2WS00JCAngWnn322cGQ/nvd/FU8wGEIVAgR/FCtDmPeLLxMCJTBgzMPIwhA5557rnvIx5sBcdv/MCZPCFXML4WQxkNRVRgPgWeeeabdf//9znsOoYGH+maBAAGf6LCvuDxQDn7k4wnhvXR4mD3yyCNdGeKOia4rVFZ4nXDCCe5BC683HsIRIRCvECMZQho2PG86d+7sPN+8NyFiKWIG0a2TWCnYxJ2HwDWUl4e1J5980u1C+cgzokNY3KEMtAuEWDxB2Q+xhQcn6iyp0d6Zm5JjeDhFkIcfQsFRRx3lBN5CaXXo0MEJenhMIdCVSpCk3CeddJINGTLEiR+8QMA4B/O+Fhp6DJNTTjnFtQ3qmnQwhsoiBPHHPrQZvLRYHx6ejXiAqAYTP3SbNg0XxBKGQPMHQ+aT5AEbUYaXCVUhSFa2P3Xt2tX1W+bA9IF+uK4gjDBnbrh9OVAF/qOdnHbaaY4r109EWYzrxC677GIIiKU0BFSiRuOh7edWRchHdNttt90qCOe83KFt0p8oL4JLIUGysowpb5o+xXWUuVDvuusu18Y5nnIeccQR7hqVROCjjSIk85IFwYlrNfXKtYDrhvfCI+18TNK0D+acRfj084XSFhAVmQsZAT+pIbzxoisqzsGQazjiLP0wieUrW5Lj0+7DtSPNvYf0qZdmwT2Ulz1R0ZH7CnXFtYM2kMTS1D19gd89eM76ERi0P/oOfZXyVIdRxjPOOMO1ebzfeUFEP6Z/8jshOlohTX9Km/8ePXq4IdncQ/39AUbR3w5J7yVpz6/9RUAEREAERCAJgUWCH3eJpgvEW4k513igTONlkCQT2icdgTvuvMdeeXWC3dl3/lwJ993/oI15bpyddOIxtkkQyObzz7+w3hdcbnt362R77rGrG55978DBdvhhB9l22/4r8PwaCEUnnHimbbVl84yY6QXJW/53bU5B8osvZtt5518aeFDuZl27dMxk/vvvf7BeZ19o66+3jp14wv/ZL8GPr9NOPy/4AbqqnXn6ie5BgubG32VXXBe8Lf4tEEFPc55wXpC8oHevjCD5/gcfuiHhl07qnzlHqRd6bLeX9Tm0d6mTLav04I23Ew/kPDBU1w/zMASuH3gJcH4/TCi8Pbrcu3dv5yXGA3pVG4I9ASvwsuDBs5AhlCFU8ODFXHLw5XgeOoq5NqYpKx5T8PPDenPlNS3vXOmkZZMrneh6hpnCkaHzeMLkMjxKGKrJfSfffrmOD6/Ha4c2SGAD6iqtIZ4Ue2yhc8HZ5y1J/4hLjzYII/pYZY02jXcPlmR+ysqcr9T9ibzwooOH/+i8q5XJJ+2QfkWbTXKdqMy5fFul3Reaa49RCNR9vn2rgjHl8/lM0i/IA/wQYIo1Xm7hKV6oXgsxSdo+KB/3TtiGhc9i81+K4wqVrRTniEsj6b0n7thSrEta95yL+uWaWpm2Voo8c/9iLm9+WyT53ZWmP6XNH/kgD0nE4FLeS9LmU/uLgAiIgAhUH4Ej+p5lw19/uspO+PyJA9zvJ35D+T9+1/LHb2n/KQ/JKquC6kuY4c1vvfWODbxviF104dkVTvzpp5+5dQiFYVsqEKhWWmlFY97GNPbJtOlu92gUb4aMr7LyvxMPE6iGISKNgiA1zB+J8aDLH+eeEcw7yQ+wXLbuOmtZs6ZNrCoFyVznrk3r+RHKW/EFaVxw/MTzhfLBkCLmOsNLrDoMESdp3uLyA18eWIuxtGVNep40vPPlu7JscqWN6JZEeEM49EOMc6WVdD03vsr0g8ocWyiPcE7ilZsvHYSyUhltuqqFyFx5rUx/8mniTcZfKa06eaRpq/zAS9s2S8EYtmnyWQqhnJdqSTxdCzFJ2j4oX2XuDaVsfz6tQmXz+5X6M+m9p9Tn9eklrXv2L3Xf93lI+8n9K03fTNOf0uYlTYTxUt5L0uZT+4uACIiACCx8BCRI1oI6R9zr0WNfu+XWO4Pho6Ns21bZATXmBPNG8vC//PIVH1jr1Vsq8FT6KRUF5qHElg7SjFpYZOC82NSpn9hMJ4oiRrJmvii50ooruDl1GAIXZ7NmfRmUJz4yWtz+WlezCTDfG38MH+Uh0E+qX7NLFZ/7hams8QS0VgREQAREQAREQAREQAREQAREYGEmIEGyltT+Fi02c3M5jn7q2SAIw2pZpWoYeC4yRPKbbxjW9a8oSZRsvCfXCTwR0xhzUGIffzLNDccOH8uwED+civNibdtsFwwdb5fxjsRDEs9I/8cwrjgbOmyEG35uxY/wiktW68qUwG233eaGN+HdcMghh1R6iG6ZFtNla2EqaznXg/ImAiIgAiIgAiIgAiIgAiIgAiKwYAhIkFww3KvkrAccsK990JuIssOy0l9jjdXdd+Zk3GH7Vplt06fPcMFjokO5MzvkWGi6RhO35YPJU6z1Dttm9pr52ec2+8uvM4Jk48aruPkC3nlnkhMkMzsGC32CCN1/BqLk8UFAmzhDtJwVDCVfa6017bk5H8TtonW1jAATrTNkiUn+GXJcroZgytyRxcxD6MtUU8rq86tPEagqAqXoT1WVt9qSrhjXlppUOURABERABERABESgdhEoHMGhdpW3VpemUeC5yHyS3373fVY5t99um2Ai/kb2yPDHbNL7k91k/x9/Mt36Dbg/GMq9lO2y87+RepkHEhv/8mtBMIn5Q66zEgu+rL56Y9syCIQzYcKb9syzY4PgB9/bJ0F6d9w5wAWk8fszfHv3drvYtOmf2pOjnw28334IAjfMsQcGDbXJH06x7Vq19LtW+GSOKzwtp0+bUWGbVtROAkQxJRJnOYuRkEc0bRYEtKnMnH81pay1s6WpVOVEoBT9qZzKU455EeNyrBXlSQREQAREQAREQAREQB6StawNtG2zg40fP8GmfvxJpmQETDj5xP/a7X372XXX/y+zfuUgoM2Zp5+UFVyi+eab2rhxL9s9AwdZh/btsqJoZw4MFg49uIcLWDNo8FDjDwGx01572vRAfCS6obeOHdoFXph/BGLoSHt42KNudYMgCmyP7vtYy5Zb5A1q0y2I4D18RDCH5G8+NX2KgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjUdAKLBENjXZiRQgVhnr+ZM2e6aLL169cvtLu2lyGB+cOgvwy8FL8JgoasaCsGQWUQEuNszpxvrUGDZdyQ67jtft2s2V8a+zYNhoX7gDa+SfHp/wicM+PTmbZocL41giHfSyyxeNYckswnSRvjb968eZk/xM22fXr605X8s8d2e1mfQ3uXPF0lKAIiIAIiIAIiIAIiIAIiIAIiIAIiIALlRuCIvmfZ8NefrrJsPX/iAKclLb744plPRkLyx+gd/ykPySqrgvJLGPFx1VVXdn+FcscQ7ySGlyV/hax+/aVtww3WywiUCJAyERABERABERABERABERABERABERABERCBhY+ABMmFr85V4gVEAG/Rl156yd5//303j+fOO+9sG2+88QLKjU5bXQTGjx9vn332mVHfjRolE/rDeavs8eG0tCwCIiACIiACIiACIiACIiACIiAC5UBAgmQ51ILysFAQ6N+/fzC/53hXVoSpH374YaEo98JeyLffftveeOMN22qrrYoSJCt7/MLOX+UXAREQAREQAREQAREQAREQAREoPwISJMuvTpSjWkjg888/d2JkgwYN7OSTT7bGjRvXwlKqSCIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQmMCihXfRHiIgApUlwJBdrHnz5hIjKwtTx4uACIiACIiACIiACIiACIiACIiACNRoAhIka3T1KfM1hcAvv/zisrriivkDADHPZJqh3EQk//HHHxNjIGr5d99954ILJT4oZkfSSBKYaO7cuW6+zJgkqmUVPL///vvYc8EiKWs4//TTT7HpRFfCJWm6/lj2J68yERABERABERABERABERABERABEVgYCGjI9sJQyyrjAiOA0HTuuefan3/+6fIwfPhwGzlypK2xxhp22mmnZfI1c+ZMGz16tAt4wzH16tWzjTbayHr06GH169fP7Pfmm29av379bI899rA5c+bYyy+/bIsuuqjddNNNmX3iFiZMmGCjRo1ywVUQzOrUqWOtW7e2zp07W926deMOyax78MEHbdy4cXbSSSfZa6+9Zq+88or9/PPPtthii9l2221n+++/vy2xxBKZ/Vl455137Pnnn7fJkycbwt8KK6xgXbt2dfMosv2LL76wyy+/3HmMHn744azKWO/evZ04d9FFFxmR4b0NGDDAXn/9dTv99NOtSZMmfnXW55AhQ+yFF16w4447zsaOHevy8fvvv9syyyzj8tqtWzebOnWqPfTQQzZ9+nRXL2xjPWWJ2ltvveXKPmXKFCOdhg0buv322msvxz28P6Lzvffea++++64rM/W2++67h3fJWv7tt99sxIgRLo9fffWVqweCHB100EG29NJLZ+2rLyIgAiIgAiIgAiIgAiIgAiIgAiJQmwhIkKxNtamylB0BxL527drZtGnTnFC17rrr2jrrrGPLLrtsJq/ffPON3XjjjYagtf3229tKK61kn3zyiSEifvTRR9arV69MMBTERISxJ5980vA+RJhD7MtnTzzxhD3yyCMuXQQyhETSfuaZZ5wn3xFHHJHvcMM7kHP27dvX7bf11lu7716soywIdN6IIn7rrbe6fLVv394Jp4iYd9xxh/OWRAhdddVVbamllrKJEyc68dELjzNmzLDZs2e7pBAMmzVr5pO1SZMmOdFu9dVXz6yLLiB+ktebb77ZnYPI1qSHSAgzWBMkZuWVV3b18vHHHxti4z333GOrrLKKrbXWWpkkEVVvv/12Y95P0llyySXdsY8//rgTNZkL1OcbRldffbUxV+iaa67poqdzLtgjPEYNgfq2226zDz74wFq1auXE4VdffdUFv4HBBRdcUEHkjaah7yIgAiIgAiIgAiIgAiIgAiIgAiJQUwlIkKypNad81wgCCJIdO3Z03oKIYng9IlB6Y5hunz593LDrU0891RAsvSFcDho0yIYNG2ZRL0KOO+eccyyfOEc6CGV4XiKmnXHGGc5TkPXkge94POKFmcQjD69NjiEtbIcddrCrrrrKENK8IInXJkIbYiP58/uy/eyzz3Yegdtss43z0Nx0002d9yHeod7jEZETwRThlWUvSCIqMkx82223zYiALhM5/ttggw3smGOOyewLe0RKhFhE30MOOSRz5KOPPuq8VhFNvSCJxyICKmLrWWedlRGQ4XbXXXe5dF588UXHgIReeuklJ0a2aNHCjjrqqIz3JHOHXnPNNU5szpwwWBg6dKjzhu3SpYvtueeebhNiMfU9ZswY11523XXX8CFaFgEREAEREAEREAEREAEREAEREIFaQ0BzSNaaqlRBaiIBhLZZs2a5octhMZKytG3b1nntIfhF54lEDCwkRpIGw7kZ+szwaIYme0Mo3WSTTdxXvPqSGMO7vcDI/oh3DEvGE9DPf4h3JB6BiH7hfRH2EGMZjo73J+bPzzHe8F5ETCRtBElvH374oVtExExibdq0yYiR7L/hhhtm8sNw97Btttlm7uuXX36ZWU2e8EBFJAx7s8Jzn332cVzxMPXG0HmsU6dOGTGS76uttporD8the+ONN5wH5I477hhenRk2Hi571g76IgIiIAIiIAIiIAIiIAIiIAIiIAK1gIA8JGtBJaoINZcAQ4axqBjJOoYDsx7BEuEyLCguvniyrouAxnGIa+PHj3fp4GnIkGGGRGN4IyYxPBfDRv4QJAn2QnrkyYuNv/76qxuuHN7fi5YImBgiIccwbBnPw6+//trwlkRMZNg18zwiEjKEnbkoKQuiZjHGeeCAWBod4s6QbIzh3t7y1UujRo1cGuQNduTLzwHZuHFjn0TOT4LsfPvtt0aAI+a5jBrzcXpG0W36LgIiIAIiIAIiIAIiIAIiIAIiIAK1gUAyVaM2lFRlEIEyJEBwF4xgKXHmRUgEKoZwF2PMvXjnnXe6QDQIZghqiF4+0E4xaeY6hiHKGEFwcpmPeo2XJoIr82QytByvQEROPBb5jiDJOsRKPCQpP0PBq8O812jYOzJ8XkRMBEk8PlnGg7VQBHV/PKIrhojJ3J5xRroyERABERABERABERABERABERABEaitBCRI1taaVblqBAEvROJlGGdemCIISzFGNGwCszB8+vzzz7ewBx+BXJgHsZTmI4Izf2QugS4ckZth2wyPxrOS4dpNmzbNiLPMK8m6zTff3BAxd9lll1JmNW9aiLZ4kFIvXhQOH0C9wBTBEhGVfWCdxDyjLbfc0g4++OAkh2gfERABERABERABERABERABERABEahVBDSHZK2qThWmphEgsjPmhwiH888QZ9Yz3DjJfJHhY/3y1KlT3TDlrbbaKkuMZLsfQu33LcWnLw8ehngzxv1RHm9+TkgfUbx58+Z+kxEghvwTeAfz+2Z2qMIFLwDH1QtD3hlejniKGIkhvhIlnfWFzDP69NNPnagZx6i6PEEL5VXbRUAEREAEREAEREAEREAEREAERKAqCEiQrAqqSlMEEhIggAsCFYFrZsyYkXUUUZ8R9pg3MSziZe1U4AvzG2LM0xg2PBIJrIIxPLpURgRtzknk6qjHIOLeww8/nDVUHOHPz6XIfIxRQRLRlCjhyy+/fAVBtVR5jkundevWjvmoUaMqlGPEiBFu7siwQNqqVSuXDOUOC70MyY+yZ6j6Flts4YZ8hwPjkAAMHnzwwcz8nnF50zoREAEREAEREAEREAEREAEREAERqOkE/nVVquklUf5FoAYSIFBM9+7d7eabb7ZrrrnGdtppJ0OkQzBkHsZVV13VDjvssKJLxryLyy23nAsWc8UVV9jGG29szPPIvJI+iAtzIbK+FIYnJ3M+IuQR3ZsANcyxiBiJ6Ir4ynY/bJlzMmx7zJgxrtyU1xvDy2FBQB8v+PltVf2JSNq+fXtDfIQbUc3xWmQI+XvvvWctW7a0XXfdNZMNooo/++yzLnAQHpTwxFsS0ZcAPVHbf//93byYiI/UNcI0w8BJH2EaDnhgykRABERABERABERABERABERABESgNhKQIFkba1VlqlEEiDbdq1cvu/fee503IF5yiHh4G3bq1Mnq1atXdHmY5/D44493aSMKIn7hoYcoyCeBY4jiXUrr2rWY/x6TAABAAElEQVSrE9QI2DJ8+HCXNHMs4v140EEHuWHK4fPhaYggGfaO9NsZto24iWhZ3dahQwcX4XvYsGHOs5PzIxTutttu1qVLl8xwbdbjwXrGGWfYXXfd5ebExCsSvoiWRBxHrAwbc4cyp+f999/vREiGrONZutpqq9lRRx3lPCjD+2tZBERABERABERABERABERABERABGoTgUWC4YV/JykQEXmJDsvQybB3U5JjtU/tJRDXfPw6PnP9Ibr5P9oWy3zyxxBi/4cXX9s+PasMYI/t9rI+h/ausvTTJjx37lwXwCVXQJi06YX3J0AL8xyStp/7MLy9KpYZto0gt8IKK1RF8tWWJuWgLfogRPlOjEckQXiScqbt401JgBxETJkIiIAIiIAIiIAIiIAIiIAIiIAIVBWBI/qeZcNff7qqkrfnTxzgnHZw3PF/jA7lD0cc/ykPySqrAiUsAukJ1KlTxwlZ6Y8sfAQvEqr7ZcLSSy9t/NV0S1MGRMWVVlopcZG5IKfZP3HC2lEEREAEREAEREAEREAEREAEREAEypSAgtqUacUoWyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQGwlIkKyNtaoyiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiECZEpAgWaYVo2yJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQG0kIEGyNtaqyiQCIiACIiACIiACIiACIiACIiACIiACIiACZUpAgmSZVoyyJQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAK1kYAEydpYqyqTCIiACIiACIiACIiACIiACIiACIiACIiACJQpAQmSZVoxypYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1EYCEiRrY62qTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQpgQkSJZpxShbIiACIiACIiACIiACIiACIiACIiACIiACIlAbCUiQrI21qjKJgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQJkSkCBZphWjbImACIiACIiACIiACIiACIiACIiACIiACIhAbSQgQbI21qrKJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJlSkCCZJlWjLIlAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArWRgATJ2lirKpMIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIlCkBCZJlWjHKlgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAjURgISJGtjrapMIiACIiACIiACIiACIiACIiACIiACIiACIlCmBCRIlmnFKFsiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUBsJSJCsjbWqMomACIiACIiACIiACIiACIiACIiACIiACIhAmRKQIFmmFaNsiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtJCBBsjbWqsokAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAmVKQIJkmVaMsiUCIiACIiACIiACIiACIiACIiACIiACIiACtZHA4rWxUCqTCCQm8LfZ338H/8lEQAREQAREQAREQAREQAREQAREQAREoLYTKBMJRIJkbW9oKl9eAn/MnWszZszIu482ioAIiIAIiIAIiIAIiIAIiIAIiIAIiEBtIDDvjz/KohgSJMuiGpSJBUmgUaNGC/L0OrcIiIAIiIAIiIAIiIAIiIAIiIAIiIAILFQEJEguVNWtwkYJLFG3jjVo0CC6Wt9FQAREQAREQAREQAREQAREQAREQAREoNYRWLzOEmVRJgW1KYtqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCEiQLItqUCZEQAREQAREQAREQAREQAREQAREQAREQAREYOEgIEFy4ahnlVIEREAEREAEREAEREAEREAEREAEREAEREAEyoKABMmyqAZlQgREQAREQAREQAREQAREQAREQAREQAREQAQWDgISJBeOelYpRUAEREAEREAEREAEREAEREAEREAEREAERKAsCCxeFrlQJmoEgWfHjLU33ng7K6+LLrqoNW68ijVdo4lttVULW3zx4prUvHl/2vARj9mkiR/Y2uusaR077J51Hn0RAREQAREQAREQAREQAREQAREQAREQARGoHQTkIVk76rFaSjF79lf2weQpNnfuXLNFFnF/P//8i40d97Ld1W+gXXn1jfbtd98XlZePpn4cpPOS1Vu6nq291ppFpaGDREAEREAEREAEREAEREAEREAEREAEREAEyp9Ace5s5V8u5bAKCfTseaA1XnUVd4a///7b+Hv6mefsoaEj7Lrrb7YLevcyPCfT2Befz3K7d9+/mzVq1NDmzZuX5nDtKwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUEMIpFONakihlM3qJbBI4C252647ub9Zs760V159vUIGZn72ub3w4nh7+eVX7euvv8na/tnnX9hX/6z7YtZs++qrr7O264sIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDtISAPydpTlwu8JDvv1NqeHP2MTXj9Ldu2VUuXHzwdBw1+OBiO/WIwv+QSwbq/7Y+5f9jOO7e2zp06uH2GDRtpUz6a6pYHDhxizTffxLp128v++usvu++BB916/ScCIiACIiACIiACIiACIiACIiACIiACIlA7CEiQrB31WBalWG65Rlav3lKBh+NXmfw8MeppNzdkp73aW+vW29qSdevamCA4zkMPj7D69evbzjvtaMcde2Qw5HtMENTmCTu/95lWt24dN2R74sT3bfLkj8yWyySnBREQAREQAREQAREQAREQAREQAREQAREQgRpOQEO2a3gFllv2GzZc1n766WeXrV9/+y3wmHzW1lyzqbXfczerv/TStthii9kuu7S1Jk1Wt5fHv5o3+0sutZS13mHbvPtoowiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQM0iIA/JmlVfZZ/bub/PtdVWa+zySaAaInITpIb5IzEfBGepJZe0GTM+dcOy3YaY/9ZdZy1r1rSJXTqpf8xWrRIBERABERABERABERABERABERABERABEaiJBCRI1sRaK9M8f/fd9/bNnG9tq61auBzO+fY79zl16ic289PPgmUicrNqfmTulVZcwX7//XerU6eO2y/6HwFyHn308ehqfRcBERABERABERABERABERABERABERABEajBBCRI1uDKK7esj3/lNecB2TTwasQaLtvAfbZts53tuUe7jHckXpIErPF/f/75p9sv+t/QYSPs8yACtzWMbtF3ERABERABERABERABERABERABERABERCBmkpAc0jW1Jors3y/9fa79vjjT1mTYLj2Jhtv5HLXuPEqQWTtxe2ddyZVyG2fm263G4O/XIZoOWvWbFtrrTVz7VLj1zOcvbZZLnG51OWMO8/IkSNt3333tauuuqrUp1N6JSBAvVA/1JNMBERABERABERABERABERABERg4SYgD8mFu/6LKv24cS/ZMsss4479PZgzEuHwjTfftgbBuuOOO8pFyWZjvXr1bPd2u9jjT4x2wW222XrLYE7JP2z0U8/a5A+n2CEHdc95/kUWWcRWWGEFmz5thtl8R8uc+9aUDfPmzbPBgwfbkCFDbNq0aUHwn58CwXUta9GihbVt29bat29fU4oSm88rrrjC7r33Xrvsssusc+fOsfuUYuXMmTNt7733thVXXNGGDx/uAiWR7uzZs+31118P5ixtVIrTKI0SE5gyZYqrnz333LPEKSs5ERABERABERABERABERABERCBmkZAgmRNq7EyyO9TTz+XycXiiy9mzZo1tY4ddjcEx+WWa+SGZvsdOnZoZ/Pm/WGPDB9pDw971K1GuOzRfR9r2XKLvEFtunXpaMNHBHNI/uZTq7mfP/zwg+2333724YcfukIsscQSttFGG9mXX35pDz30kPs78sgjrVevXoYYWxPtvffes19//dUmT55cpdmfNWuWffXVV/b999+789WvX79Kz6fERUAEREAEREAEREAEREAEREAEREAESktAgmRpedbq1Hp03zsQEvfOKiNDq/PZYostZt267mW77bqTzfh0pi0aiG1rrNHEllhi8Swxcued2libHXcwhuLiSYg1abKaHX1UT3ugz3Pue03+7+STT3Zi5Prrr2+nnXaabbvtts6DlDKNHz/erbvjjjtc+c8999waWdSbbropGJ7/jrVq1apK87/VVlvZgw8+aA0bNjSJkVWKWomLgAiIgAiIgAiIgAiIgAiIgAiIQJUQ0BySVYJViUYJ1K+/tG24wXq23nrrZIZ0R/eprd+/++47GzNmjCveddddZ7vssktGjGQlAh7rsWHDhmUJtW5lDfmPodJt2rQJ6rduled4yy23tLXXXrvKz6MTiIAIiIAIiIAIiIAIiIAIiIAIiIAIlJ6APCRLz1QpikAWAebOwxZddNFAkF0va5v/0rJlS1t66aXt22+/tbfeesu22GILv8l9fvLJJ27uyY8++igYFr+cbbjhhta1a1dbdtlls/YLf3n55ZeDQEOPB5HKP7emTZvaZpttZnvttVdmzsXwvrmWCbyDNyKejwyTXm211QwPRdKhPGFjPsepU6e6+TB9/hFj7777bufJeNRRR9krr7xi48aNM5isscYawbD9ltauXTuXzM8//+yE2wkTJmTyTBkZ2h42hrkPHDjQ6tSpE8xZelx4U97l999/32BCWShXs2bNHMN11123wnF4q/744492wAEH2IwZM2zUqFH22Wef2cUXX2wrrbRShf2jK5LU16uvvmovvPBC4C28hP33v/+tUC/Mxwlz8gfvsKUpS9++fd18pYcffrhRH0888YRNnDjR1Ql1yZyOtD28namb1157zT744ANbZZVVrHXr1pn68ednqPxdd93l5uo87LDD3DEvvfSSq3uYbr/99k6Y9vsn/UzCLJwWDGhzzMcKwzXXXNP22Wcf167C+2lZBERABERABERABERABERABESg/AhIkCy/OlGOahkBRBrsr7/+shdffNF23HFH9z38H/NGvvHGG04UIjJ52IhKfNJJJ1XwnEQUYph08+bNw7u75d69ezvRLrqBgDo33nijCwgT3Rb9Pn36dDvooIOcEBfedt9999ltt93m0l9++eUzm8jnM88844ZSe0ES8ermm292wunXX3/thKzwMH/K8H//93/Ws2dP94cQFrb+/fvblVdead26dcusRpAkTYZrJxEkOd8tt9xiN9xwgxsSn0koWEB4PPPMM+2II44Ir7YBAwY4URSR9J577skcx3D7QoJk0vpCnD7++OOd6EhZEPe8IQyef/75TqREdPNWTFlgyLybtD+Wf/nlF5+cE7mHDh1qd955p51zzjk2YsSIzDYWqOvu3bu7QEV+A/Ohwp+gU4i1sAobaSF+nnXWWRVE1vB+4eWkzPwxROy+/fbbs+arZRvrOC/tSSYCIiACIiACIiACIiACIiACIlC+BLJdnMo3n8qZCNRYAkSDZpg2duKJJzqPQ+bKjBpeXnj9hT0P8RZEBCOq+bXXXuuiFDP8G5EIjz08637//fespBDZ8CDEs+7++++3t99+23mSIRIyX+UFF1yQtX+uLwiFnKNtEAGc9PBuRBzE25LANaeeemquQyusR5h84IEHnMj25JNPWr9+/axLly5uP8TNPfbYw3kkIg5SPoSx3XbbzQmBl156qSEMFmucC3YNGjQwIoFTjkcffdSJZtQDZSIgT5xx7Oabb+7q4Prrry8oRqapL+bAxOMSI228ITG8NxEjMeo37CFambIgyiIwE0QJvkcffbSbOgAeeEI+/fTT9p///Mdte+qpp+yEE05weRg0aJATy92X0H8IzGw7++yzndclaSJEYnjFcp4kloYZ6dF+aDMI9/AbO3asm+rgwAMPtD/++MMuueQS10+SnFv7iIAIiIAIiIAIiIAIiIAIiIAILBgCEiQXDHeddSEjgEcXnowIc4huBLVBGGR4LJ5rcUZwH0QjBCqOYfgy8zQiCF522WVuaCqeb4iO3hDWEN3wuES0YX5KxMxNN93UeUaynuHHkyZN8ofEfs6cOdOJjgQlQhDcbrvtnFflvvvua+edd547huHGYW+72IRCK/FQPOSQQ5xQylyTzJu5+uqruz0YSnz55Ze78lK+TTbZJCPKMYyd4bnFGsPW69WrZ6eccoqLdI5AvPHGGxvBg9ZZZx0nej733HOxyVNPgwcPdsJg586d8w6RL6a+GK7OcOyffvrJ1RuZYIg1w5AJgBT1AK1MWRAbieKOMA1f2lT79u1dueGPVyFekmxjfk68chlWjzEkO87wlMS7lLxyHExps9itt96a8SyNO5Z1xTBDOMX23ntvQ4SkDSEaI07SzulPfh+3o/4TAREQAREQAREQAREQAREQAREoOwISJMuuSpSh2kgAIRGPMcS8Jk2aGN5lDAXef//9bYcddnBDZhEew4YIhxjHcN5OnTqFN7nljh07uk/mRfTGMsN6EdKYUy9szP/YokULtx1PyXzGOREj8SBEHAvbzjvvbKNHj3bCJl6dSc17iYb39/NHMicmHMLWuHFj22CDDdwqho8Xa3BHqEW8ihpemNjHH38c3eS+H3vssYmHHRdTX5wEYZqh7wQ0evjhh93wcthfffXVbm7EcMYqU5Zdd901nJRb9vw5n/duDO9EXWMMzY6znXbaqcJq5grF05djmFM0nxXDzM+bGldneJDSNg899NB8p9U2ERABERABERABERABERABERCBBUxg8QV8fp1eBBYaAgzFZp7AnoEnGgFNHnnkESfq4eWIxyNDlRlujTcfRnAbDGGRfaPmh2rjzejNH8P8fmHPSb+dYcJY+Bi/LfzJfgyjfuyxx9wcjwyvZug2npJE0cazsBRG4BRs1VVXjU2OAD6YL2vsTglXMvckXp1ffPGFGx6N2Pruu++6o+OG0LMBj9Kk5tmnqS/SRqy+6KKLDPGT4fnYMccc4zwO3ZeY/4opS0wyLnAN66lTzzq8n1+Xhr/34iVwEUP+cwVy4jzFMMNTleH/9CGC2CDMI4ziWUt+fZ7D5dCyCIiACIiACIiACIiACIiACIhAeRGQIFle9aHcLAQEELm22WYb98cwU8RGhnTj3cjwVy8kerGMT78chwdB05vfj/n8+MtliHKFjCHWDNklP0R85o9ozHjNMfR6yy23LJREWWxnWDBzMuJdyByDGKIvXny//fZbyfLo2aepL39yIl3j1UobYEg5c43GWXWVJe7cadatvPLKLpI6Ed7zWTHMGJZN5HeGhxMIij8EXbxsO3To4Dw9l1xyyXyn1TYREAEREAEREAEREAEREAEREIEFTECC5AKuAJ1+4SZAYA68vBBTmEuQodR4vxHJmWG0GENtEVpyWXjYtD+GgDTM65fLvGdiru2sJ63TTz/dzb2IUEagE+ZaJCAMcxki8hEkpdyNiOMEX4Ex5UEMXmqppVy2iVJOQJlSmGefpr78eRkW/+abb7qvBLfBk9MPl/b78FldZQmfs5hlInFjCNj5rFhmBPpheDvCOkFuaJe00WuuucZ5HRPpu1A09Hz50jYREAEREAEREAEREAEREAEREIGqJSBBsmr5KnURcENxiXTNHH1x8/SBiCArzJmIRxmBbhAgCSyC4cnHMNUkxjHMt0haSY8plC6iEfM78sf8lIh4eE8S7AYxtZy90fCAxJsOr1QiP+O5V1VWTH2RF4KwnHHGGc5bk2HxiGtErkZo8/Mlsl91loXzVca8Z6QPWpQrrWKZ+fQY6t8zmAKBPzyFu3Xr5uYLxauXoDwyERABERABERABERABERABERCB8iSwaHlmS7kSgdpDgPkcEWiIbo2gF2cMJZ4zZ47bxJBdDJESGzt2rJvz0H2J/BeNck2kYwzvsVznih4TSdJ9RbzDQ4+oy2FD2COKMkOemVdw4sSJ4c1lt/zJJ5+4wDx4kfq5OcOZZI7DUlkx9cW5+/fvbxMmTLC11lrLbr/9dhf5Gi9ZhiGHrTrLEj5vvmU/BD68DxG5CdqE968XHMPbw8tpmSHK7r777kaApA8//DCclJsP84ADDnDrXn/99axt+iICIiACIiACIiACIiACIiACIlBeBCRIlld9KDe1kABekXgRIjoxzPmnn37KKiUiC/Ph8UkwGebIw5hTEK9E9j/llFPsu+++yzpu5MiRbvgx0Zm9cS4iNhMs5Morr3RinN+GQHnJJZc4MWfSpEl+dexn8+bNjWHEDHV+5513svZhOLEXNaORvLN2LIMvzIFJMCEimA8YMCAj0hLE5tZbb7UhQ4a4XPohxpXJcjH1BWOGGWPMJ4pwSiR2hjpTr88880wmS9VZlsxJCyzghRgOCIQQiecshrciAW7yWVpm9COmG0Ccve6667LOTftmWgEMcVcmAiIgAiIgAiIgAiIgAiIgAiJQvgQ0ZLt860Y5qyUEiP47dOhQFzl54MCBbg7GzTff3A0f/vbbb11QDrwj8Si75ZZbMvMbUnxEqh49etiLL75o7dq1c1GuEasQN4lizNBs0vLWoEEDJzoiFPXt29eeffZZa9mypRM1GQrOsFYiZTdr1swfEvvZokULNxz7/9s7D7gpivOPj9Is2HuL2LHEij0g9oqiRsWKPWrUgBrrX8QWKzFoTNRYUQG7KGoUG1bsJbaoKHaNxq4goPnvd/A5511273bvvbv37r3ffD5322ZnZr8zO7vz22dmmAiGGbaxvESkRMg0gZJusvU+ozGsmIAHK0QELLrydu/e3fND6CX9sH/zzTcTOeTdmSe/6KrNmJYI0TBGnMPRrfzII4/0FpJYqPbo0cN33a71tWS5drqXb7DBBj7tCJOI1YiSWPkya3gWl4cZ4Q0cONCXw3vuucdPrET8WO7a+KuIoGlDI2RJj/yIgAiIgAiIgAiIgAiIgAiIgAhUn4AEyeozVgwi4CdUGTVqlJ9ABTERgdEsyxgjkgltDj30ULfMMsu0oIWYSVdvLBtZ2szZWEEiUA4ePNhbjIUn0aWViWeYAIWuqya2YWGHOMf4hMRZzCHwMPM3k4cg4iFEvvTSS/4UJgs58MADXf/+/YsFUTfHuF6EWgRaBFl+iFaww5IOJnSpnzhxYgsxuJwLyJNfiKTkD+NExrvG77nnnr7bPczpuj1kyBCfnFpeS5brp0wfd9xxhSECGG8UAZUxRhHLs7g8zAgPYZx4sSRlhm3KOo4yu+mmm7qjjjrKEaacCIiACIiACIiACLR3Ao+9/qz75wsPuVc+eMN1mLGDW2qBxd0+G+zollmw23SXPuZfj7hRT9873X7bsdzCS7rDNt/Lbz41/kV31UM326HU5epLrOT27f3b1ON24Icpk92wh2924958wf3n6/+6ZaP0rRGdu9M6W7lOHVpKEs9OeNld/sANbv455nGDdjjMgvDLv425xr3y/jRDgj9suXfideLx7/de615+7w1/zoCt9nFLR1zMXTfuDvfwq0+53yzXw/Vbbxvb7ZcDrz7dTZk6xR259X5uifkXa3Gs2MZ7//3InXXbxX5S0HN2O9Z17tgp0fs7n33gzrn9Hy2Ozdx5JrfoPAu6Xt3Xcqt1W6HFsWIbIYskf33W2NhtvnJPf+jaR0e5x19/zs0x62zutJ2O8O/Ndg75ccpNF/jNPXv2dWsvvaodarG85L6o59y7r7ktVunltll9oxbHwo0wXeXmUVJZ7dKps1t8vkVcz+XWTORk8YbXHaYrXE8KPzwe3gu2P08ZtnO0LE1ghqibW/KgdrFzEU/ef/993x20a9eusaPabFYCScXH9rFM+2EdZj/KFuss+U2dOrXwY4y63ufvXTW8u67Xx53ff1DVwk8LmPEX33rrLS+MIfDRrbiUgyX3IEvExSwOnsSDgNkaa0a6aNO9GMs3xsRE/Gk0R7l69913vRhbarKVSlxbOfmVNd5aX0uYrvfee89bRbIPsZuyS/mALeUyaazO8Pxi6+UwI1665JOn9TzBUrHr1jEREAEREAEREAERyEvg1Jv/6i64Z5hvG/BuznsUrkP0bnbWrse4/r12aBHk+Xdf5TgnzW2w/FruxgEX+sM3PHGXO+Ty0m2kvj02df844E9pQfr9CF5bnrWve/ezD/32jDPM6H76309+ff1l13BXHny2m3OW2Qth3PbMvW6/S46LxNVfuXGn3FTYz8ou5x/u7n/5cb/voE12c6fuNLDFcTa+mfSdW+Gozd2kKT/4Y7cddYlbd5nVCv6OGXGWu/zBG91ePbd3Q/Y4vrCflUUPXd8hPN159GVuzaVWbnGs2MbZt1/izhk9TWi84qCz3TarbZjo/Zm3X3JbnLlP4jF27hwJtH/de3CmtlbIIinA47c72A3cal9/6K3/vOd6ntzPTZ462Q3d60S32/rbFk457KqT3cjHRnveDw0aEYmp0xuufPfD927FP27pWC6/yNIOf2kuTFe5eVSqrPZbdxs3tP+JjrJkzuI9oe8hbsCW6YzxXyr88F7Af94yzDn17vaP7rFR0b1WLTf2D1f5HqD0ArUfxiv8aD/asuXniGqlRuGKgAi0INClSxdvNdliZ4kNXjQWWyz7lzqC40aPW12WiCbxMCIT1pKN7KgIazm2YDn5lZVvra+lVLooH3SFb60rh1lWcb61adP5IiACIiACIiACIlAvBG5/9j4vqmBdd+7ux7mtV+vtPvnqv274o7e5of+80h038hy3TiTCLbfQEtMlea2lVnEX7jN4uv0zdepS2LfVqhu4p077ZZz6qx++1cdHmBcExhyzdpmlcE7ayrEjzvFi5K8XW86dvdsxjuW4N593g28a6h59/Rl3cmSdd96eJ6Sdnrr/xif+6U7a8XDXMbIMDd2op8cUxMhwf7XWEYJHPj66EPyIx25PFSTNE2l+/JQb/eZn33zh7njuAXfxfSPc9ePudOssvZrDUjGrO3iT3ROtVOec9ReRd8nI2vPwLfZy546+1J1264UOK8LZZprVYY163eN3+KjO3PXoRDGSg7c/c78XI1l/9YM33QvvvOpWWXx5Nou61uZRWFa/+v4bd9sz97nLHrzB88aK8dDN9iwaf6mDYfih3/BeYH+1ynAYZ7Ou/yIpNysBXbcIiIAIiIAIiIAIiIAIiIAIiIAINAiB02/9m0/psdv+zlvVIQwiOv3f9r/3gt+UH6dGItJ9iVczSyRidptv0el+C845X8E/4YV+5u46pz+GABrun2/2uQvnpK08Of4FfwhruR5L/trR9RYLtAFbTLNio/usWXemhRHfT7yfffO5u/dfj8YPuRGRtR8uS9qmO7mMHYiqdNlG3ILTfS896i3qigXFR3jjCBOE1Y1XWs+fMva1J4qdOt2xubvOUQjLwmQZWp1yErzZ/+nXn7shd1zqmR8/8ly/xNK19/JrTxe27Rj+2G1+datVe/slomspV4k8CssqAuiJOxzqdv25m/0tT91TKgklj4fhh+zCe4FAqlGGSyauSTxIkGySjNZlioAIiIAIiIAIiIAIiIAIiIAINDYBBKXxn7zrL2KXdbae7mKGHXKuuz3qprxtZAVXD87Gb7zz+QdaCI+k762hD3pLQQS6PG7b1add28jHWwpjdE1GPFr5V91dt3kXyRNk2X5NnNtl3a3ddmts4n6MhiLD0jGv6x5Z/OEmR0OWVcMhBJ/Z748+aMaDPOO2ixxdyLvONEti13dLw4RP33ePv/GcWygSrLGipJv0TU/e7bt/m5+kZbXyaL1lV/fRvR+JwLVy1SjDtUp7vccjQbLec0jpEwEREIE6ItCpUyfffZ8u/HlfHuvoMpQUERABERABERABEWhIAhOiiVFwCEnzzDbXdNew6NwL+u7ayyZ018bzx1996ru80s3YfrdG3Zyr5Q7ZdHc/cc0dzz3oNjp9D9/l9svvv/bvkXQb5pfXrbX0Km6RuRZw97z4iPv82y8Lp1vXacZInPjzGJKFg1VY+XbS9+72Z+/319cnEkl3XGtzH4uJlFmj/HritwURc6OV1s16mvf3QjTRjOWjLZnsKMlhhcmENFjQnnfn5d7Lsdse5OIWgeG5I3/u0t23x2ZelPzNcms48u+u5x8KvU23Xq08gjdu6YSJm6ZLRIkdWe+FapThEklrmsMaQ7JpsloXKgIiIAKtJ7Dgggu60aN/GSen9SEqBBEQAREQAREQAREQgawEPvnyU+91rlnnKJzCBC4X3Tu8sM0KXVDpiht3r334ljvsypNb7EbYTPLbwlOZG5tFMz1f8/s/uxNvOM+99N7r0Xh8Z7vBNw6NxLstonEN+/uu5nmDxopwl2hikz/feZljnMIDN+7nJ8q5PhLPsAREGPzH/SPzBpvb/6hnxriJkyc5rnGuaMxGum0jCL/+0dve+pCZxJMcVpSWB59Ek/48+u+nI4vDKX5cRrMsTDovad/oSKDjF7ptI0tNsyQM97N+4vaHFvwvMMe8bv8Nd4l7KWwz+dB1P4+PaWLrDhHbh157ytGNe7semxT8xlcqkUfvff6Ru/Cea3zQX038JuoO/5if6ZsdB0dDALTWZb0XqlGGW5v29nK+BMn2kpO6DhEQAREQAREQAREQAREQAREQgXZNoOvM0ywKv/thYuE6J07+wdm4krYTa7gkkXH5hZdyh0VCYOhmikS8arqNVlzX8UPIGvbwze7uFx521z46KrKye9DdOPBCP+5lnvi/j0TA3dbv486763I3Iuq2jSD58GtPuw+++MTtsObmfvxEhMJqu+GPTusyvsOam/mo6D20fbR+wd3DorEsb3dpgiRCn1lzWhoZV3P4oeelTixj/uJLxMctVunVYvdikSia5mwSG45/8tVnbuyrT/i8SfIP0/c//9iLxjaJzTarbeSOHn6WP++jSBynK3eSq0QeMTQBkx+FDhGVGcSx9Gyty3MvVLoMtzbt7eV8CZLtJSd1HSIgAiIgAiIgAiIgAiIgAiIgAu2awGqLTxs2h67KiG5MNDP7zF3dgydOs5C8Npppu5h1IILOTmtv2SaMenVf0/Ej7ftefKyfZRtx665jpnUfzpoornvxaIzI9ZZZ3YeB5aV1k6a7Nq7agqSNV0lczEKOpSbuP5HFI45JV07f+Uhvsel3BH/Msj3u1Jv8HsTZE64f4rDWw3Iyr/v1Ystmzs83P3nH/fWeYT6KtZde1T0RzXZ+TGSx+shJ1yWm08bohOWuFwwoJK1zx07u20lT/AzdA7bcu7A/XKlEHq0UXdsxfX7ng+0SxbnYvAv7fO/UoTIyVjn3QqXKcMiqmdc1hmQz576uXQREQAREQAREQAREQAREQAREoGEIzDHLbG6Zn8fPG/3cAz7dHWac0a246DL+t8Ac89TNtSAUbnDqbm7zM/ZuIRAyG/VR2+zv0/nCO6/67sp5Ej31p6neu4mPF98/wt0RsVhsnoVcz+49/LGpP/6YJ8jcfk0A5URm2r43ml2b34vRmI44xoW8I5rIJ8lhSYmgym+/DXd2Sy3wK2+tGO92n3Rua/Yh/tI1HOvZkYcNdfPPPo9j0prz7rpiumBJ/+hnp6UfS0i7PpaMnYkzwXK6k6MdlcijebvO5a0/sQDdMLKwZXKZSomRSWlO2letMpwUVzPukyDZjLmuaxYBERABERABERABERABERABEWhIAtZF+IxRf3dY6oXuk6+mWeiF+9pqvfsiS7kPoi6/z0542V3zyK0tksEMz7jZI4EVi7tyXJ+o2y6T4ox8bLRjHM1+0biSzAJdbUeX6+vH3eGjufSAM9z4vzzQ4nfEVvv5YyN+7tJdLD2IyUwsg6OrdzhJT7Hz8h7DgvPhqMs8rLHcZFKkQTse5oMhXpu53cJloiOYYqUYv75Xz73bzdplFn/OU+NftFMSl22VR4mJKWNntctwGUlqV6dUxta1XSHRxYiACIiACIiACIiACIiACIiACIhAfRJA8GK2bYS4jU7bw627zGpunsjq8NUPx3sLPUSuDVdYJzHxz0cWidsNmdYNNvSA1eW5ux8X7mr1Ol2TmUSFiWyOv26I+3s08Q5pffWD8e6l91/34R+08a5lx0N3dcZsHPbwLX7W7l3X61N2WOGJR484y3eDD/d16djZXf+HC/yusa8+6T784j8Oa9UtV91gOkG137pb+wl3GDOTcS2ZEbyY2y4aBxIrxVfefyM673J32s5HFPPe4tjVkdD7wCvjWuxjAy5799rR7//q+2/coBvP8+u/i3hjkYnbee2t3JVjb3JPv/WvqOv2We7GARf6/fyNjMbAxDH5EEMCxN2Wq/by3dSHR/7WXGrl+OHCdrXyqBBBbCWNB2XbLIvtlCz3QrXLsKWlWZcSJJs153XdIiACIiACIiACIiACIiACIiACDUeALr/n7fl/bvF5FnH3v/yYezASpKb+9KOfzAXB74S+hzjGCExyX37/tXvs9WenO1StMRf799rBLbfQEu6M2y7yYun14+70cdO9+tDN9nL79v7tdGnJs4Nu2wiSPZfr4bts5zk3zS/ddONupk5dCrusuzYTvCRZdy4x/2JutW4ruOcmvOJnqTaLyUIAsRXyk4la9rjwCHfF2BsdoiF8srh3P/vQ8Yu71butWNjFhEeffv25Y8zEMC3Ee0a/P7rNzugfTVLzpB/3EiHzjY8nuKcikZLjSRMjETBCJVaXtz59j/vTLkf6sUwLEcZWqpFHsSgKm2k8vpn0XcGPrWS9F6pdhi09zbic4X+Ry3LhP0ZjMLz//vtunnnmcV27Tq+QZwlDftofgaTiY/tYpv1+igbstR9li3WW/KZOnVr4TZkyxfU+f++qgeMr2vn9B1Ut/HoP+Pbbb/fst9tuu0JSk/YVDiasfPzxx+6hhx5y//lP9JVwjjncEkss4ZZbbjk333y/zLj29ddfuzFjxriFFlrIrbfeegmh1Meue+65x33zzTeub9++rkOHDvWRqJ9TMW7cOPfBBx+4jTbayM0111y501bP15b7YnSCCGQk8Nxzz7m33nrLrb/++m7BBdNnnMwYnLxViUCjPCMqcfk//PCDGz16tK/Hqc8bybX2OdRI1/ryyy876o8vvvjC1x282yy//PJu1lmnzW5c6lqaiVUpFlmPN1M9kJVJHn/f/fC9H9cPwameHW1DZm3GsjDJ6q6e0660iQAE2ksZ3v+S49yoZ+6tWqaO/cNVrmPHji1+tK/5zRhZcNuy+gMsVO0SFbAINA6Bf/zjH+68885zCKxt5f7+97/7NFCJmnvwwQfdfffdZ5t+mbSvhYdg48UXX3SnnHKKD+Oll15yjz76qLv66qvdww8/HPhy7ttvv3UIYrzc17N77LHHfDoRyEu5r776yp166qnu1ltbjodT6rxyj7/wwgs+bbywl+PyXFs54VfrnFdffdWddNJJdV92qnX9ecOtdbnMm75a+3/ttdf8ffPZZ5/VOmrFl4NAozwj0i7p7rvvdieffLL/MJfmx/ZPnjzZl8knnnjCdjXMsrXPoUa5UJ7r559/vn+XeeWVV9y9997reI97++23M19Cs7DKDCSDx0avBzJcYlW9MJ5fvYuRAMDiDss/iZFVLQ4KvIoEVIYrC1ddtivLU6GJQCKBN99803355ZfeGjH0cMMNN3ir0H79+oW7K76OwDZ+/HgviNIY6tLlly4HrYls+PDhPv0777yzW2uttVynTp3chAkTvKVklnARLt955x235ZZbeuvrLOdUwk8luGNJicWiLMYrkSPpYWCBy+/DDz90q622WrpHHfEE6qVcIoxibb3IIou4DTfcULkjAu2aAM+9jz76yD/n559/fn+tugcaM8s/+eQT989//tPNMsssbu+993YrrLCC++677/y7ypJLLtmYF6VUi4AIiIAIiECdEpAgWacZo2Q1BwEsJBAIqy1IYhZ92mmneUG0UmIkAitdmeadd1638cYbFzKse/fuhfVSK1i/PfPMM65nz541FSQrwX3RRRd1Z599dubuW6VY6HgyAcSsVVddtaxu6skhtu+99VIuv//+e29d9Otf/1qCZPsucrq6iMABBxzgsF6fc845Czx0DxRQNNQK4jI9SXjurLLKKj7t5GuYtw11QUqsCIiACIiACNQxAXXZruPMUdJEoJIEZpppJv/Fv1JhYv2BW2yxxSoVZGo4WCcwtmgWRyNw0qRJWbyW9MOYpgivYTf38KTZZ589daxJrFIRbMvtps/5ebtn4z8trWG6WSddcK20Y3w08iCro4tWKUZZxsxMskBOSkOW+JLOs3158zVr2YUB5YXwS7mJEyc6u/+S/BYrl/injOQtW1he8vGk0o77mjwp1+XND649a1kph1O515F2Xtbyk3Z+0n7uzzz3Pv65r5Mc5TBr3Uz5yRNvPD7Op+yXcnnLRKnwshzno19rBCueWa1hkyWNoR/yM0t8sMxTV+CfuiKLs3uRZdxRH2atF8qpQ4rVAVa3/upX02agjactbTsvK8LJ88wu9T4Spou8TbtnQ3+s4zfrPRw/l23Kbta48J/nOvAvJwIiIAIi0L4JyEKyfeevrq5OCZxwwgn+pZ0GFi/jhx9+uE8pYxYlucsvv9w9//zz7rjjjvMTw5gfxgUcOXKk22WXXfykDbafCWawiFxppZXcgQce6Hcff/zxXvg555xzzFvZy2OPPbbQmGGcJNI/22yzudNPP91h9ch4lUwiQbqS3JVXXumeffbZghB17rnn+jFlDjvsMLfMMssUTmHcpqeeesp3lWK8DhoIe+65p8MKzBzdr+n6vc8++/ixLOkev+KKKzrCirus3GmQjBgxwv373//2aZx55pkdE/+EXU/pnnfGGWe0YEx8NHRuvvlmb/nJS76le8cdd/ST/cTTFN+m8c84nP/617983HQJ33zzzePeCts0Bm677TbHeJ6ffvqp747P9e+xxx6J1pukm2vDCoQGFBauv/3tb1t0h/7888/d4MGDfV7EOT755JPummuu8d3s6Wpvju7rV111lXv33Xd9mabb4q677uoYW+29995zf/7zn82rXz799NOe03//+18/sPFSSy3l0/ynP/3JTxxw8MEHe3/kLXm8/fbbF/hbng8YMMCXDyxeaVQxODKTJlHuGD4gdFnjC88J1/Pma5ayS/jkA9fDkArUBVzDyiuv7POEvAkd7EeNGuVsPEQ+MvTq1ctts802hWEY0sol4TAxHWO5co/SEKZLIt0Ryadw6AHGer3iiit8mcei+s4773TkE2WZ+5NujExwl+To7kg9YCIDE0NQP3BeWJaoz8jbN954wzdmEXPIuz59+vjykBR2uC9vftAIpq6krHCPFSsrWTgZZ6yo9t133zBpbtCgQf76GV8XZua4P7AI/+Mf/1jyQ06p8sMQBtTl1E3ER1kw95e//MVP5EO+rrvuun439/odd9zhr596AsckZFtttZXr3bu33+YPMYl6EsvWHj16uJtuusnXKwhu1LvUswsssIC78cYbfR1OPnCN+N99991biHLUj4hK3MvDhg3z9YDVOQzzYdZnhchTVqjbxo4dW6iPuS+oD0hf6PKWCTuXSdm4np122slb69t+ODFO8Oqrr+7LvO1neeKJJ/r6mfoKNvaMJm87d+6c6R4gHCZcuvbaa/3wH9wzXNtee+1V8lnBOZRl7q2ll16aoAqOsSy5X+19gjzg+c9zYbPNNvPxUcaLxZf3OUR5pK7nuUVdzPsAPR/C+9nSQV1A3t9yyy3+XuS9plu3bj79jENNPcdznOenldENNtigxb2E56x1SNbnBc+ws846ywtmhE+ZII0MFUK55znLfUn9R5kwl5dV1mc29w/5yP1N3VHsfYS0UMcxTAbjeZO/OIbM4PnOZDxxV6qOifsPtyk7PEvIA55hbFMvkN/x+zLvdYTx8Nym7kiqZ3mH5F1yzTXX9O+F4XlaFwEREAERaBwCspBsnLxSStsRARqANAwQTWgYs84vzS277LK+0c4EDaFDOODLNC9moePFlf2cZ47tPF+x7bykJcLc2muv7Q/xEkraaTDgaHAST7Ev7ggunGOz3hIW23PPPbcPgz9erGlIIIjssMMOXjhhvMkzzzyz8LKNP+IhPl5MOc4M38yGmeSycicOREm6otPQRjhGzHj99dcLwfICTryhdR/rNAJpKCL0kG5epGnosJ/GZzHHtSAykJ80/hGaiP+uu+7yL/7xc2mAXHTRRe7+++/3jVLiY3ZzzqehHKbNziV8BIQtttjC/eY3v/HXefHFF7coQ3ZtXHfcESfXzdIc4gwNOfjT0KRRwozqF154oRfa8B+6xx9/3E8QgPhBo5W0EB5pjjO1+MLyZHl+ySWX+MluGL8UMYuygsjF+F+hyxNfeJ6t583XrGUXbkx2hWDLPYDIQtnlvkbApWFvjjJ12WWXeU6US4QkLCFpFF566aXmzTcM4ww5SMN26NChXhCjgY34QhmlwYfognWmObuHx4wZ4+9Bxk3bdNNN/f3JPcC9luaYgZZ7mfzAIbCwHTZSEZgoczSamWWY6+beR/hETKP8FXN584OwEPlpPMO5WFnJyonyjRiI4BqmF0EeUZaPQtwPoWNyDMpo+EElPG7rWcrPwgsv7KjPSC9CiTnKCYIz5Widddax3e6CCy7wMzsj8FBPcC75zMcJBDlzdu8jLCEEkPfcn9QrXBv1GD+EI4bnIBzCJE/xHzrKIemjbkDw3nrrrf1Yw+zjoxX5UcrZBy54UuapLxDTmWCEe91cOWXCzuU6SCt5GTo+trGfdIb1HWnhBxvESBzx4xemWe4BzqGccP8j3vFxB1GOjw3U6YhcxZzVgcQXd6SDnznLU8R/4iMvisWX9zlE3UHdAieeKeQTXLif+bBhLkwH40+TDp4X3Ec4yhRliGcv6aOegztlNCzj+M1ThxirUs8L8oG6yt6bKBdsm3Bu4YRlIS8rzs36zCYPEXF5LsJkk002SX0fgQnseFeAK/c4dS5ljLzhPgpdljom9B9fR6jlwyvXs+222/r84h7gviTs0OW9jvBc3n9wPBPj7yO85xC25U94ntZFQAREQAQah4AsJBsnr5TSdkSAxj2OGa15yUJ4KuawdMTxUmlWeryIsd2xY0eHUMlXd7OSQZDE2cuc36jgHxZ7NLZpECIqlkp/PGqsC/hhVcGPRubiiy9e8EbDZvTo0V4wwQIEAY2GAS+lvMzTODnkkEMK/llB0DjmmGMKDFoc/HkjK3e49e/fvxAEL/nMuklj3xorhYPBCtwRHzn/97//feEIlgiIqwiHxQbFx+IVHghGWLZaY5cwsSKNN1KxXqIM9O3b1zcIiJC8QTx94IEHvFURjZjQIcZgOWUOgWbIkCGeKfGGFl3mp9SSBgjlkUZQaM3J9WDZQRk1RwMOKz/iGThwYIEHFqj45ZysDgu/o48+upDnNIYZ1xMLG0RRXCXiy5OvecouQin3LdazlG8c4g9CLg1uGlzcGzjKDg5rM7tXaPhj+YRfGp58HEhyCAGISAjARx55ZAsrZKyrKC80MOOWfnSPhS9WNjjK2RFHHOGFeazRkrqo0himPkBspX4jTWH9gNUZjVZELCyjWOK4fgRXBFKECfIyzeXJDwsD8bZUWcnLifucOhBh1YauIP/5yIRQxDoiE478gRkWi8XusTzlB4EP/9zrWKRTB2LVheiJJbnFMyGyYEIMpe7Bgs8cggX1CudjaRs6hAYskK2+QjRGGKHeRxxBxDYLXsov5QJBjzKGsGMOptyLVr7Zz/MMi0LqAcQES6edY0sstanvEayw2rTnG+FR7nkOUH9hkVhOmbB4sObmRxik19IDW+ou7gOEeLMywx/OnssWji1L3QPmj3B5zph4Tx2IIE+djiDMtVXSkTdZ4sv7HKK+omxzf2HpjuOe58MePRx47obdn3lWYAVKmTXHPYTlPWI94VCGcTChnuAZzEdP6pxy65BSzwvqCNLNRx7uF/LX3rcsnfFlXlZ5n9mUR8pdqfcR6tpx48b5epN73xwfQfkggKiLxTYuTx1j4YRL8hRrWPjwPLLnOwIyz1+EaOoWPtqYy3od5t+WlAM+6vLuxT2xxhpr+EPUT1iCIv5j+SsnAiIgAiLQuASmfdpt3PQr5SLQFAR4CedFnUaRWUTwwswXaSwJEF1C6w788TKY1q2y3qEhPOKwnrLGIdtYVvICyoupcWA/DrHEGqzT9pT/H+9qbsKudZUtFTIvy6GjIUXjPi4Ohn5YR6DCYXFgYiTbCEJJkwUhWGFlGxcTrIFLwyPuEA1Dh8BK2AhI1s0rPF5qnTJIOhAhaJCEjnSE4gTHKJtY1JCXJnbYOaFoYfuKLWmshnlOeIgBWGDRAMJVMr4s+VpO2Y2HS9dXygvWvnEX+iXvEXXxy3WnOcQwZiqnYRcOiYB/LNz4qICIi2AROkRBEyPZT8MTq0qcdfv1Gzn+EFv4CINwbWIkp1Pe6VrI8r777ssUYsiCE4rdZ1nKSl5OJkhxTeawquN+oiyG9x/lEGd1ifmPL/OUH/KD7qM4BAc+mmA9iHhoYiHHyEOER8pK6BCjEWEoG3a/2HEEpPj9yQcLHGJDGD6CoNVPSeUiLuogKiA08PGFeifNwRXBHuEqvM8pN5RDrvXtt99ucXqeMhGeSL7wwQcrUBz3At2GEdPgHOYlgiTPJcv/MJw864jYVlfbedSLuKzPGjsvyzJrfHmeQ6QTC28+kpgYSVoQ5RFUKVfcE6EjHaEYyTE+qvJM54OMiZHsZ50PL5Rhyimu3DokSx3gI8jxl4cVwZbzzM7yPsL7IGXSPsTZJXAP06WZZwX3Ei5PHWPhhEu7F3iXMDGS4zyDiJ98RCCNuyzXET+HbRPmGe7CHPUpgj4ftilrciIgAiIgAo1L4Bezlca9BqVcBJqCAA0mrASwUKE7Hi+FNCZ5gacBz0s/DT0a1Viw2EtcI8KhkYkwQUMnLpJxzXRlxXombBRX8qU0HhZx4uJdhuJsEZDoSknjgK7RNOCxAKK7o1nXxM8Jt2nM0wAjjFKOPEbYI+ywy6WdRwMEYS7uQqHTjtGYpEFI49IsvexYqaWJf5wXNk7SzrOGdlq3+rTzkvbH84kGGY0iurkhTJCeSsSXJ1/zlF3uUSzssPSirHOPI0jwASJufYioc2XUVZqut3RRxyoEUYayUqq82FABcTESpjBjP4196o5QQI7zxb8dL3Uv4DfJFUsLkxdxT9MdlkZtUlklzDz5YWmIX0tSWSmWtiRO3NOUMe4dxHTKGvUVwihWYFgrci1mfcf1mKBr6Yov85QfzuXeJW6GKaBLLnnZOxKZQ0ddwI86A+tT7lkEN8Qi0hn/uBOeG67bkBphvWvHs9aR+IcZ6cTCCWZp5dfERoQHnn2hMwHV6rhyykQYHvcez1HyEqZ89CIOur1joc7ztV+/fv5+QQxBsLVrDsPJsx4vk5xrYZZ7fxWLP2t8eZ5DlkdYH8bziPKGszrY0paUjgkTJvjDoSWl+edDXvgxr9h9WqwOicebVAdYnFmXeViV+8yOpzupjMCPujn+3OA69t9//xaXk7eOaXFytMH55Hf4scr82DPGxGPbzzLLdYT+bZ16lmvmnuS+4AOIiaI8C+VEidpkggAAHzpJREFUQAREQAQam4AEycbOP6W+iQiYIBk2mPg6TOMOEYNum4gw1p0M/43osLijEU9jkC59aQ7rmKSGcZr/Wuyn0c+EP3Q352s+3bT5IdZh5UF+pTlEAUQCBMYszoRaGkRYRiU5GGVxWJ3irAGZ5RzzY1Z1JlTZ/rSlpSmr/7Rwsu6vRHxZ8zVv2aVBR3dEJhvh/qXMIFphbYa1YNjgQxihUUb3f0RMLFCwGqNMMR4kDcQ0Z1ZoSY1VzrG8QNyJT5CRFma5+7GKw4XWkWFYXCP3P/mWlt6s+RGGm2U9Lyc+HtAAtwk4aCQjclAfY7WOIMk+BENELNjaeHlJ6clbfiwMwqebKXUIH6hIQ9whFlEv4bC2J88RCEzYi/uv9jYfJBAki9U5CIE4ynuas/NbWybIR/IT6zusd8k3hhvAepgPS9yfWE9yzxGnDaWQlq5G3V/ucwhu/JKc1cFJx2wf9zzO6iLbn7SsRB2SFG7efeWyqsQzO0wrHxUok6XGpuWccusYi4+4eE6E3bHtGEsTS/lgXCnHMxErTz4YUGdwP/KBgOeDCaCVikvhiIAIiIAI1J6ABMnaM1eMIlAWARpwiA689NM1CitBumDiECUYPwgLGRq+NJrC7lNlRdhGJ9Gw5As418BsoWmOxmM9OtLFmGr8sHrjqz5jtDGJBzNfpzVkeemmMRZOZFLs+qyLLlax4ZhRxc5JO4ZFIc4aE2n+kvabsMS1ZnHm3wTVLOe0xk+l4suSr+WUXaznmMGVjwncuwghWLEx7ld8Nmas6/hhMcYQDYzlxRhm3PcnnXSSt4JLYmXCnuVz3I8JBmljUMb9t2YbCyasvElLkvhAWrj3Ld/S4sqSH2nnpu0vhxPddqmTsRqikYx1nYXDhwj2YSWNYBAf0iCejnLKD2FgYYs4ghDJOmWE+sQc6eOjBVaEv/vd7/ywF3aMsRnjFmx2rJpLq+eK1TlWx5HGtA81MDPXmjLBhz0+BGDdTtpYMmQIjvyDJ/cm5RfXqB/8fOKL/JX7HKI7rs3oHg8+LIvxY7bNPWP1QrGPK/ivVB1icZe7LJdVJZ7ZYZop99SZafV76LfcOsbCIC7yJy0ue5bwXKukozcBgiRjDDMUEb1DsJpN+vhSyXgVlgiIgAiIQPUJ/PLGWv24FIMIiEArCPDySzfN8ePH+/HeeAG1cbtogPKiSIMJC0m2491jWhF1zU/FKoUGPGMeYVGU9MvSyKl1wmn4Iwzb5DNmGXnAAQf4pCA0FXM0ujk3i0AAIxxCIGUhiVGSNRbCRdzZuGlmcUp4OCYqKOXoxkleYLVijZFi51gcWPrWwlUivjz5mqfsMlA/DSwc9ytd05hwCCs3LFEQHHFYoVKuECxx5Cvj+DGhAGOAYnFjx7yH2J+VFevqGB7GQo79CDJZLGzCc8tZN9EzKS2UN8o+ol6xhmae/MiTxnI4mTBFPmIpaR+JiBdLHupry0fzWyxNecoP4VDfjx071pcDxhNG1InPMm/jC9OAN2voYmmoxTHSiSs2zrHlB3VLWv1GucVVokwgLmNBxiRdLC0vYcZ9Zs9XhHTKaFs6nve4LHV03nSW8xziA1NaHllai6XDBGezig39IvaPGTPGDynB/krUIWH4rVkvh1XeZ3aW9CEA8s6UJBQybiX8KNO4vHVMPH7480xKek+hvsNVYkiWMF7uN6wysZBkvGMcVpNyIiACIiACjU9AgmTj56GuoIEJ0NDhJRHxIYujQUtXQIQtGk/WGOOrN2Il1lKIQlkavlniq7Yfs4AxaxmLzywtrrvuuunGN2O8RBtI3vznXeblnjV8RKErrrjCd58MzzFhMEkMDP3RLRdHgzjsSklX0riARyMPy1i6umE5EDrioau4NfrDY1hQhQ5hCFGDRgYCKg4+iJKEbWO0sZ+yGu8+SRnE4gNnwovfiP5oOMQbSDTsETG5prhVpc0kbedXYlmJ+PLka56yS1lmxmnElNBZebEygMUwM9Aym238XjG/xcoWHy5ohJIfJj5bfIiiCD58xLD6xI61ZkkZwsXTi4Uw8SCaxY+ZpV+p+itPfuS5hnI4cd8gSpCX5IGJWMSLIEke0p0a4S1trMQwjXnKD8+NYcOGeTGbiZAYFoK0MAQAeWrOPt7E6xC6/ZuowHOlWg5BJHTUNwgXCAzhsAShH9axiiLt1IfxskK9dfPNN3vLYvxWokxYuUPgxXIzFFXIS5hi8c6ztphgTnpwaffAtKOt+7cPLfG6gwlLWttdNs9zCKtb6nOeyRN+HgfSrox3kWuvvdaLV7YvbUm5hyn3SlgWsRxnRnby2iwnK1GHpKUj7/48rMp9ZmdJE5MjUdfEx/FkXGCeMXzQMmviPHVMUtw2SRV5Ys8o/FEfMcM292xrJ3xKipf6gDjocYIA261btyRv2icCIiACItBgBNRlu8EyTMltXwR4aWNMuCuvvNJ3se7du3dRUQDRgJf2eMMXKohTNPw4Xo2XwWqQp2FH44/uhFhGMPYaDXw4IG7RuDr77LP9l3CuiwY1XSB5oaYhwL5yXF7uWeMgXY888oi3RmBQd/ILQc+s4OIzi8bD5TiiHI0HLF/gg2BAviaJ1nSToyGO+IgVCYIKjUAYITzR4A8teeAFb1hjjYdfExiZ2TvkybXw4g9/GjuIkVha2Th7Ydq33nprP8Ya6UBkxNKOxqnN5hn6xRKQuCjzQ4YMcczkjDBNwxqhotKuEvHlydc8ZReLNfLt0ksv9UIWAggiMsIhjUezAKEhy0Qp1BV05Ub0QuAi78lrRKhiY2nBgAk5/vrXv/rZlmlQcp8RN/lPOaHbeCUdgg4CN+Xgpptu8mIcHEkrwhni45lnnunzH1GV68D6hWsOJ7BISlOe/Eg6P21fuZyoT7hXYApLcwiQ7EMUMOHCjqUt85QfBAHqB3iaNeFuu+3mhg4d6u8vxrNFHOCDAWUHARJrWiYOoezQ1Z/jPE/4+FBMHExLb5b9V111lY+L8g0L6jjqmnidEw+LesQm7DnllFP8PUC5QozkHuGaOU79UYkyQVdgGFA/0k07rA+550aOHOnrYRMu4+mNb6fdA3F/5WxjIc1zEyGQjz7c/9Qd3EetdXmeQ3wsocz97W9/83UL9ZQxZExcJiEhvKThGcJ0ktecSxnlmYP4xLnkM2UTq3ELoxJ1SBh3a9bzsCKevM/srGmDHR+quc8RpHn34FnNexT3OPeauTx1jJ0TLplIhnJH/lL2ePckr9iHaM/HkW5VEAuJl3HFEakp/3IiIAIiIALtg4AEyfaRj7qKBiWAGEMjAsGJH9vFrJR4IedFD7EpLjrSSOJcGgM0hBrBIYohQCBcwYHGOz8agkcccYQXLRBMrr/+en85WIVss802/hc2FvNea17uWcPnq/3xxx/vrr76aj8GmQlyjI9Fo41GQzFH/h199NHusssu8wIdAixiFAIN4wbGLQgJl7EDhw8f7huiCJ80PigDBx54oG8ohPFhYTJw4MCCxQnHKFOMQRl/wd9uu+28ZQuNDqwe4I3gSWOG+EKHCHPUUUf5cGmU4AgXkYtGPF3vQ4d4QHhM+kFXMhwN0v3228+Lc6HfSqy3Nr48+Zqn7GJ1NWDAAF++sW5GzMYhGDEGaTjjLBPXIEJSBu6++27vj7gIgwlwrJu9P5Dwx72GQEXZtAlQqCdo+NNYNeujhFPL3oVQjWhGfFwL+YBjP0xvueUWf5x9lKFNN93U9e3bt4UQxLG4y5Mf8XNLbZfDiboXQTK0jrR4yB+sQeP1tR2PL7OWH+uqTZ2IIGkOIYJ7mbqAeDnGRwnqA+5FJmfhR93BeJJ8/KArMh9QqDeq4Q466CCHtbt9/KBuQCBHyCjltt9+e182EN9skjPOhzVj8lq5r1SZIJ8QJON5CS8EVQR2GGd1afdA1vPT/JEe8pT72fIUi0wEL6x145bQaeEk7c/7HKL8H3PMMf65gLU+VnMIld2idxWee6FInxSf7WO4Cp7/PG/smU++InD16tXLvPlla+uQFoG1YiMvq7zP7KxJ47nPmMN8FETE5X2SfXwU4v4LxxTPWsekxc35hx9+uH8/o6xRFxEX90dSXqWFk3c/zz+ug6ExNLt2XnryLwIiIAL1S2CG6MXhf1mSxxcpxojhgWDdLLOcJz/tm0BS8bF9LNN+WGTYj7LFOkt+dNexH1ZZvc/fu2oQd12vjzu//6CqhZ81YL5oIzxZF6+s57UXf3TH4ws7L+u87MYdfLAYoyFaSVdN7pR9rJFoMJPupOsqdi1YRDImFNYgWc7lHsJaislAKEulHGNVInJSpxdzpIMB5PFnXb6K+WdsKe5frI2wbDnxxBN9IxMLpyRHmhHD+CHI0cCm8UnDphqutfHlzdesZZe6zvKvlDhI3sGZrpJZ8iTOkXvNylb8WDW2KQdp5ZJ7n2vn3i/H5c2PPHHUmlNS2rKWn6Rz4/tgZWMO2uQscT+V3B48eLC30sIyl3JKmYVpqTonLQ2UFeos666c5q+aZSItzlL7i90Dpc4tdpxrpYwgCFUjT/M+h7iXSQ/PLdJUrqN+4rmX5eNqa+uQctMYPy8vq7zP7Hh8aduUCZ4lvHeYYJ/ml/2trWPoBcIzy4YPKRZXa48NGjTI1yW8V8iJgAiIgAi0jsD+lxznRj1zb+sCKXL22D9c5Y2l+HhnP3oj8eMdwZaykCwCUYdEoFYEEBaa2SHEFhNjq8WnWuGSlzSmsO4o1yEq5jmfij2PfxMBS6WPdFh30DS/NAixtGJW2lA0xlIDF3YbZ5uuwogTdNEPxQXzH1oG4r+1rpLx5c3XrGUMwSarJVHWvEvjhvUSgkGtXLFyWey+z5K+vPmRJUzzU2tOFm+4zFp+wnPS1mFVDdEqLb74/rBuiB/Lsl3qOWFhVLNMWBx5l8Xugbxhhf651nIF3jCctPW8zyHqMawcW+v4gJHVtbYOyRpPKX95WeV9ZpeK345TJvLU762tY6pZ/uyaWPJ+wJAP9B6QEwEREAERaD8EJEi2n7zUlYiACIhAmxBAjKSL3SuvvOLHq0PIofHAmJOs0/07dHQlZawzLCHpIos1DJOrMDYc3UYZs7KSrtbxVTLtCksEREAEREAEmpUA7wX8eI4jdtukOs3KQ9ctAiIgAu2NgATJ9pajuh4REAERqDGBjTfe2AuPjPHGxCTmEBcZ5y20guTYAQcc4Me6YrxJJtkxx7htjCOGCX8lXa3jq2TaFZYIiIAIiIAINCuBiy66yH+0xBJ2r732KmuYkmZlp+sWAREQgUYgIEGyEXJJaRQBERCBOifQs2dPx4+xq/jRZYyuYHQfizvGEUF4ZAIGZuVkXDi6hbe2S2c8HtuudXwWr5Yi0MwEmHSFMSO5/+REQAREoBwCjCdN93Ymk6r0x8py0qNzREAEREAEKktAb4mV5anQREAERKCpCWANGbeITANCI4PZtWvlah1fra5L8YhAPRJYeOGF6zFZSpMIiEADEVhllVUaKLVKqgiIgAiIQF4C5U+Blzcm+RcBERABERABERABERABERABERABERABERABEWh6AhIkm74ICIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAI1I6ABMnasVZMIiACIiACIiACIiACIiACIiACIiACIiACItD0BCRINn0REAAREAEREAEREAEREAEREAEREAEREAEREAERqB0BCZK1Y62YREAEREAEREAEREAEREAEREAEREAEREAERKDpCUiQbPoiIAAiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiUDsCEiRrx1oxiYAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEDTE5Ag2fRFQABEQAREQAREQAREQAREQAREQAREQAREQAREoHYEJEjWjrViEgEREAEREAEREAEREAEREAEREAEREAEREIGmJyBBsumLgACIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQO0ISJCsHWvFJAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJNT0CCZNMXAQEQAREQAREQAREQAREQAREQAREQAREQAREQgdoRkCBZO9aKSQREQAREQAREQAREQAREQAREQAREQAREQASanoAEyaYvAgIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgArUjIEGydqwVkwiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAg0PQEJkk1fBARABERABERABERABERABERABERABERABERABGpHQIJk7VgrJhEQAREQAREQAREQAREQAREQAREQAREQARFoegISJJu+CAiACIiACIiACIiACIiACIiACIiACIiACIiACNSOgATJ2rFWTCIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiLQ9AQ6Nj0BAWhqAlN+mOw+/vjjpmagixcBERABERABERABERABERABERABEWgOAlMnT6mLC5UgWRfZoES0JYEOHTq0ZfSKWwREQAREQAREQAREQAREQAREQAREQASaioAEyabKbl1snECnLp3dfPPNF9+tbREQAREQAREQAREQAREQAREQAREQARFodwQ6du5UF9ekMSTrIhuUCBEQAREQAREQAREQAREQAREQAREQAREQARFoDgISJJsjn3WVIiACIiACIiACIiACIiACIiACIiACIiACIlAXBCRI1kU2KBEiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0BwEJEg2Rz7rKkVABERABERABERABERABERABERABERABESgLghIkKyLbFAiREAEREAEREAEREAEREAEREAEREAEREAERKA5CEiQbI581lWKgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQF0QkCBZF9mgRIiACIiACIiACIiACIiACIiACIiACIiACIhAcxCQINkc+ayrFAEREAEREAEREAEREAEREAEREAEREAEREIG6ICBBsi6yQYkQAREQAREQAREQAREQAREQAREQAREQAREQgeYgIEGyOfJZVykCIiACIiACIiACIiACIiACIiACIiACIiACdUGgY12kQokQgTYkcMHdw9owdkUtAiIgAiIgAiIgAiIgAiIgAiIgAiIgAs1FQIJkc+W3rjZGYMRjt8f2aFMEREAEREAEREAEREAEREAEREAEREAERKCaBNRlu5p0FbYIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiEALAhIkW+DQhgiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIQDUJSJCsJl2FLQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIi0IKABMkWOLQhAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiJQTQISJKtJV2GLgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAi0ICBBsgUObVSDwAwzzJAr2Lz+cwUuzyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAlUhEGo64Xo8MgmScSLaFgEREAEREAEREAEREAEREAEREAEREAEREAERqBoBCZJVQ6uA4wSKKeOt8Rs/V9siIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAJtR6CUBiRBsu3yRjFnIFCqAGcIQl5EQAREQAREQAREQAREQAREQAREQAREQATqiIAEyTrKDCWlJQGJkS15aEsEREAEREAEREAEREAEREAEREAEREAE6p1AFj1HgmS956LSJwIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIALtiIAEyXaUmY16KVmU80a9NqVbBERABERABERABERABERABERABERABETAuVD/kSCpEtGmBKww2rJNE6PIRUAEREAEREAEREAEREAEREAEREAEREAEKk7AdB9bSpCsOOLmCtAKUjlXHT83vl1OmDpHBERABERABERABERABERABERABERABESgbQgkaTvxfWxLkGyb/GnXscYLWtLFFvPDsWLHk8LTPhEQAREQAREQAREQAREQAREQAREQAREQgfogUErbkSBZH/nUVKkoJjaGx0oV3qaCposVAREQAREQAREQAREQAREQAREQAREQgQYgENd2kpKcW5D83//+lxSO9olAIgErhLZM9KSdIiACIiACIiACIiACIiACIiACIiACIiACTUMgsyA544zTvE6ZMqVp4OhC254AQuZCc8zX9glRCkRABERABERABERABERABERABERABERABFIJdF9gydRj8QOZBUmEoZlmmslNmjQpHoa2RSAzgbilJNtpP0Rwju2xZp/M4cujCIiACIiACIiACIiACIiACIiACIiACIhA7QnsssYWXscxPSdN7yFlmQVJPHft2tVhIfnFF1+wKScCrSJAwcziNl9+/Sze5EcEREAEREAEREAEREAEREAEREAEREAERKANCHTq0Mn1XHKNTDGjB+USJGeddVY3++yzu6+//tp9+umnbvLkyU5jSmZi3XSeSomNacfZH/91nKGDu2bPM5uOoS5YBERABERABERABERABERABERABERABBqBwLA9/uQ6zthhOk0nVf+JBMXcs9QgSH755ZcSIxuhRNQojUnF6KeffirEbsfZxzo/W//xxx8L26yHv6lTpzr7fTPpO3fLS/e7Me89WQhXKyIgAiIgAiIgAiIgAiIgAiIgAiIgAiIgAm1D4LcrbOx2X7uPm23mrq5Tp07+17FjR8evQ4cO/se6deNmn1+PhKHcgiSXyGkTJ070YlGZQbQNKcVaFQJJZSAuSOLH9tm6LREhOWZiJPtNiGSYANZZYpU7ecpk983330Z+fyqUP87jHJzFYRdq+21bSxEQAREQAREQAREQAREQAREQAREQAREQgekJxC0aEQ9xLDt37uy6dOnys9g4o5uj6+yuc6fOfr+JkSxNkCQsEyY5vyBGsn/6qLPtIdBZZpklm2f5avcEkkS/UBjkOD/bx9K2WTdB0QRJloiQtjRB0kTJrrN29cc5l2Ms+eFYhukJ19t9RugCRUAEREAEREAEREAEREAEREAEREAERKBMAqEgyXooSLJu1o5edOzQ0hKS4+H5th3uI1lsly1IlnldOq3JCVAYEQwpfCYUsm7bLHEsQ7+2jZpOoWebH2Iky1CQJFz7lYPb0lXOuTpHBERABERABERABERABERABERABERABOqFAJpJOc50FzsfjcZ+XowMumSzP/Rv28Rr59txS4sESSOhZasIULDiQh4F0KwWk44Toe1naducR1jsYx0RknBY4sJ4LA6zsORYeDzuP37MB6g/ERABERABERABERABERABERABERABEWhSAqbJcPnhum2zj591uTatJhQmbR9L82/rFk64lCAJDbk2IRAWctYpqAiLVnBtyX4KuYmJLEP/Jlay335cUOi/TS5QkYqACIiACIiACIiACIiACIiACIiACIhAAxFAb8GFS9NnTItBmAx/ptuYABn6Jyzbb+FyXIIkNOQqQoACZSKgBRjus3WW5vBPwUSIxLFu+1jaj3MQJemizZJt/Jp/REn84uwcv/Hztq1rKQIiIAIiIAIiIAIiIAIiIAIiIAIiIAIikEwg1GxYt23TYViapaQtTacxP6bX2LbFxH4c+yVIGhUt24QAhRABkaWtm8jIthVW80MiER/jfkNBkvW4M7Eyvl/bIiACIiACIiACIiACIiACIiACIiACIiACv1hFhixMlzGNhiVCpC05Ht/mmPknLNu2dZYSJKEgV1UCFLxQULT1UCS0wml+TZQ0vxTuUGjEn3Xvxg8/O27bSRfFMTkREAEREAEREAEREAEREAEREAEREAEREIFpBNBYkpxpNRxDp7FtlnERMjzOOj9ceI7f8fOfBMmQhtZbTYCCVkr0Mz8scRRSExfZZ9tWeAnP1jnOD/GRfSz5hX4Ik+1S6Sh1nHDkREAEREAEREAEREAEREAEREAEREAERKC9ETBNJu26TH+x47aNFsOP7fg6+xAq2Y8zf+yPb0uQ9Ej0Vy0CFDqEP1sSj63bkn0UUhMZESfDbQqziYe2tHNZml/CseO2DPexLicCIiACIiACIiACIiACIiACIiACIiACIpBMAJ3FnK3b0gRGW5omw9IsJu0YS1x8m334lyAJCbmKEqBghYJgUuDmx5b4sXUKq4mT4TIuTHLM9nFOGCfr4XZSGrRPBERABERABERABERABERABERABERABERgegJoNPzM2bbtM6HRhEj2275wyfl2Trg+QyTaaFA9o6tlRQlY0bIlgbNu2wiKto91O2br4TJcN3+2JAyOh0u/Ef1ZXGyH63ZcSxEQAREQAREQAREQAREQAREQAREQARFodgJJoqExQWDE2RK/4S8UIMN1/Nh2uE5YspCEglzNCFAATRiMr1siKKwIjOGSY5xn55hAGd/POaEzoTLcp3UREAEREAEREAEREAEREAEREAEREAEREIFkAnFtBV/oMfazbfzZPpa2HV+aH4uNbQmSRkPLihOggIUiokUQ7rd9tgzFSvzhLAwTIW2bpfkPhUfbx7kWButyIiACIiACIiACIiACIiACIiACIiACIiACxQmEWkq4bkIl++I/EyHZb/4slrhfvx2JN+qybYS0rDgBK162tAhMQLT94Tb7THzEv62zP/6z4xauhcd2uG7HtRQBERABERABERABERABERABERABERABEShOANHQXLhuYmOiyBiIkRwPRUrCCs/9f8jpoQCL9bzoAAAAAElFTkSuQmCC)" - ], - "metadata": { - "collapsed": false - } - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Create a Giskard client after having install the Giskard server (see documentation)\n", - "api_key = \"\" #This can be found in the Settings tab of the Giskard hub\n", - "#hf_token = \"\" #If the Giskard Hub is installed on HF Space, this can be found on the Settings tab of the Giskard Hub\n", - "\n", - "client = GiskardClient(\n", - " url=\"http://localhost:19000\", # Option 1: Use URL of your local Giskard instance.\n", - " # url=\"\", # Option 2: Use URL of your remote HuggingFace space.\n", - " key=api_key,\n", - " # hf_token=hf_token # Use this token to access a private HF space.\n", - ")\n", - "\n", - "project_key = \"my_project\"\n", - "my_project = client.create_project(project_key, \"PROJECT_NAME\", \"DESCRIPTION\")\n", - "\n", - "# Upload to the project you just created\n", - "test_suite.upload(client, project_key)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": false - }, - "source": [ - "### Download a test suite from the Giskard Hub\n", - "\n", - "After curating your test suites with additional tests on the Giskard Hub, you can easily download them back into your environment. This allows you to:\n", - " \n", - "- Check for regressions after training a new model\n", - "- Automate the test suite execution in a CI/CD pipeline\n", - "- Compare several models during the prototyping phase" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "outputs": [], - "source": [ - "test_suite_downloaded = Suite.download(client, project_key, suite_id=...)\n", - "test_suite_downloaded.run()" - ], - "metadata": { - "collapsed": false - } } ], "metadata": { @@ -675,14 +3661,14 @@ "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.6" + "pygments_lexer": "ipython3", + "version": "3.10.14" } }, "nbformat": 4, diff --git a/docs/reference/slicing-functions/index.rst b/docs/reference/slicing-functions/index.rst index 1d718c9b20..3f5abec88d 100644 --- a/docs/reference/slicing-functions/index.rst +++ b/docs/reference/slicing-functions/index.rst @@ -6,8 +6,6 @@ Slicing functions .. autoclass:: giskard.registry.slicing_function.SlicingFunction .. automethod:: execute - .. automethod:: upload - .. automethod:: download Textual slicing --------------- diff --git a/docs/reference/suite/index.rst b/docs/reference/suite/index.rst index ca1dffa54f..8ddc58994d 100644 --- a/docs/reference/suite/index.rst +++ b/docs/reference/suite/index.rst @@ -10,8 +10,6 @@ Test suite .. automethod:: remove_test .. automethod:: upgrade_test .. automethod:: update_test_params - .. automethod:: upload - .. automethod:: download .. autoclass:: giskard.core.suite.SuiteInput @@ -21,6 +19,4 @@ Test suite .. autoclass:: giskard.core.suite.TestSuiteResult - .. automethod:: upload - .. autoclass:: giskard.core.test_result.TestResult diff --git a/docs/reference/transformation-functions/index.rst b/docs/reference/transformation-functions/index.rst index 6528ccf7a1..5ad58538af 100644 --- a/docs/reference/transformation-functions/index.rst +++ b/docs/reference/transformation-functions/index.rst @@ -8,8 +8,6 @@ Transformation functions .. autoclass:: giskard.registry.transformation_function.TransformationFunction .. automethod:: execute - .. automethod:: upload - .. automethod:: download Textual transformation functions -------------------------------- diff --git a/giskard/visualization/templates/scan_report/html/_code_snippet.html b/giskard/visualization/templates/scan_report/html/_code_snippet.html index 3cae7daf1a..4bad732009 100644 --- a/giskard/visualization/templates/scan_report/html/_code_snippet.html +++ b/giskard/visualization/templates/scan_report/html/_code_snippet.html @@ -1,30 +1,12 @@
-

Debug your issues in the Giskard hub

- -

- Install the Giskard hub app to: -

-
    -
  • Debug and diagnose your scan issues
    • -
  • Save your scan result as a re-executable test suite to benchmark your model
    • -
  • Extend your test suite with our catalog of ready-to-use tests
    • -
-

- You can find installation instructions here. -

-
- 
{% raw %}from giskard import GiskardClient
-
-# Create a test suite from your scan results
-test_suite = results.generate_test_suite("My first test suite")
-
-# Upload your test suite to your Giskard hub instance
-client = GiskardClient("http://localhost:19000", "GISKARD_API_KEY")
-client.create_project("my_project_id", "my_project_name")
-test_suite.upload(client, "my_project_id"){% endraw %}
-
-
+

What's next?

+
+

+ 1. Generate a test suite from your scan results +

+ 
{% raw %}test_suite = results.generate_test_suite("My first test suite"){% endraw %}
+
+

2. Run your test suite

+ 
{% raw %}test_suite.run(){% endraw %}
+
diff --git a/giskard/visualization/templates/suite_results/_suite_results_header.html b/giskard/visualization/templates/suite_results/_suite_results_header.html index 33c9c6b8c0..7bf6a24d42 100644 --- a/giskard/visualization/templates/suite_results/_suite_results_header.html +++ b/giskard/visualization/templates/suite_results/_suite_results_header.html @@ -1,19 +1,18 @@ -
+
{% if passed %} - + check - + Test suite passed. {% else %} - + close + d="M19,6.41L17.59,5L12,10.59L6.41,5L5,6.41L10.59,12L5,17.59L6.41,19L12,13.41L17.59,19L19,17.59L13.41,12L19,6.41Z" + /> Test suite failed. - To debug your failing test and diagnose the issue, please run the Giskard hub (see documentation) {% endif %}