{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Blockwise Ensemble Methods\n", "\n", "Dask-ML provides some [ensemble methods](https://ml.dask.org/modules/api.html#module-dask_ml.ensemble) that are tailored to `dask.array`'s and `dask.dataframe`'s blocked structure. The basic idea is to fit a copy of some sub-estimator to each block (or partition) of the dask Array or DataFrame. Becuase each block fits in memory, the sub-estimator only needs to handle in-memory data structures like a NumPy array or pandas DataFrame. It also will be relatively fast, since each block fits in memory and we won't need to move large amounts of data between workers on a cluster. We end up with an ensemble of models: one per block in the training dataset.\n", "\n", "At prediction time, we combine the results from all the models in the ensemble. For regression problems, this means averaging the predictions from each sub-estimator. For classification problems, each sub-estimator votes and the results are combined. See https://scikit-learn.org/stable/modules/ensemble.html#voting-classifier for details on how they can be combeind. See https://scikit-learn.org/stable/modules/ensemble.html for a general overview of why averaging ensemble methods can be useful.\n", "\n", "It's crucially important that the distribution of values in your dataset be relatively uniform across partitions. Otherwise the parameters learned on any given partition of the data will be poor for the dataset as a whole. This will be shown in detail later." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's randomly generate an example dataset. In practice, you would load the data from storage. We'll create a `dask.array` with 10 blocks." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:24.260576Z", "iopub.status.busy": "2022-07-27T19:21:24.259856Z", "iopub.status.idle": "2022-07-27T19:21:28.359080Z", "shell.execute_reply": "2022-07-27T19:21:28.358324Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/share/miniconda3/envs/dask-examples/lib/python3.9/site-packages/dask/base.py:1283: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.\n", " warnings.warn(\n" ] }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Array Chunk
Bytes 152.59 MiB 15.26 MiB
Shape (1000000, 20) (100000, 20)
Count 10 Tasks 10 Chunks
Type float64 numpy.ndarray
\n", " 		\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " 20\n", " 1000000\n", "\n", "
" ], "text/plain": [ "dask.array" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from distributed import Client\n", "import dask_ml.datasets\n", "import dask_ml.ensemble\n", "\n", "client = Client(n_workers=4, threads_per_worker=1)\n", "\n", "X, y = dask_ml.datasets.make_classification(n_samples=1_000_000,\n", " n_informative=10,\n", " shift=2, scale=2,\n", " chunks=100_000)\n", "X" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Classification\n", "\n", "The `sub-estimator` should be an instantiated scikit-learn-API compatible estimator (anything that implements the `fit` / `predict` API, including pipelines). It only needs to handle in-memory datasets. We'll use `sklearn.linear_model.RidgeClassifier`.\n", "\n", "To get the output shapes right, we require that you provide the `classes` for classification problems, either when creating the estimator or in `.fit` if the sub-estimator also requires the classes." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:28.362563Z", "iopub.status.busy": "2022-07-27T19:21:28.362171Z", "iopub.status.idle": "2022-07-27T19:21:28.370560Z", "shell.execute_reply": "2022-07-27T19:21:28.369789Z" } }, "outputs": [ { "data": { "text/plain": [ "BlockwiseVotingClassifier(classes=[0, 1],\n", " estimator=RidgeClassifier(random_state=0))" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import sklearn.linear_model\n", "\n", "subestimator = sklearn.linear_model.RidgeClassifier(random_state=0)\n", "clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n", " subestimator,\n", " classes=[0, 1]\n", ")\n", "clf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can train normally. This will *independently* fit a clone of `subestimator` on each partition of `X` and `y`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:28.373464Z", "iopub.status.busy": "2022-07-27T19:21:28.373014Z", "iopub.status.idle": "2022-07-27T19:21:32.128773Z", "shell.execute_reply": "2022-07-27T19:21:32.126218Z" } }, "outputs": [], "source": [ "clf.fit(X, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All of the fitted estimators are available at `.estimators_`." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:32.136476Z", "iopub.status.busy": "2022-07-27T19:21:32.136015Z", "iopub.status.idle": "2022-07-27T19:21:32.144059Z", "shell.execute_reply": "2022-07-27T19:21:32.143568Z" } }, "outputs": [ { "data": { "text/plain": [ "[RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0),\n", " RidgeClassifier(random_state=0)]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.estimators_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These are different estimators! They've been trained on separate batches of data and have learned different parameters. We can plot the difference in the learned `coef_` of the first two models to visualize this." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:32.148015Z", "iopub.status.busy": "2022-07-27T19:21:32.146997Z", "iopub.status.idle": "2022-07-27T19:21:32.825114Z", "shell.execute_reply": "2022-07-27T19:21:32.824482Z" } }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:32.829869Z", "iopub.status.busy": "2022-07-27T19:21:32.829260Z", "iopub.status.idle": "2022-07-27T19:21:32.974893Z", "shell.execute_reply": "2022-07-27T19:21:32.974310Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEJCAYAAABhbdtlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVaElEQVR4nO3de9BlVX3m8e9Dc/NCbAyNQkPTiB1LIBNHW7zMpIaIyC0INUIV1MxwicpoqamaEqUdxslldGxjjEqkZJgEwZgKRSZjbKUdRAypUYsEGEXTKKFFLi0tCAEV0RjkN3/s1fHQnnXe8/Z5mxe6v5+qU+/ee6219zqX9zx7r73POakqJEkaZ5fF7oAk6YnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchsRNLclGSd47MvzHJPUkeSvKLSf5Vklvb/MmL2NVtlmRDkiMXux+PlyTXJnndImw3ST6a5IEkf9uWbf16eijJc+ZYz4pWb8nj03PNJX5OYseU5HbgWcAjwE+Bm4GPARdX1aNj6u8GfB94aVXd1JZdA6yrqg89Xv1+IklSwKqq2rjYfZlWkmuBj1fVH3XKfwl4N/BrwG7AHcClwIeq6qczbPdXgT8DnldVPxz3enq8zfVYaDoeSezYTqyqvYCDgLXAecAfd+o+C9gT2DCy7KCt5qeWZNdtabczaXvfj9v/YJJDgL8B7gJ+uaqeAZwKrAb2mnH1BwG3V9UP2/y415OejKrK2w54A24HXrnVsiOAR4HD2/ylwLuAXwJ+CBTwEPB54Jut7o/asj2AZzCEzGbg263tkraus4AvAh8A/qGV7QH8PnAncA9wEfCUVv9IYBPwVuDets6zR/r6FOD9DHu63wO+MNL2pcCXgAeBm4Ajp3kcgN8GrmA4ovoBwxvY6gltC3jumOWT7tfewKeB7wIPtOkDRtpey7An/8X22D63becNwK2tzYW0o/zW5jeAr7eyq4CDRsqOBr7RHqMPA38NvK5zfz4OXDnH6+bV7XF5sPX1+SNl+wN/0e7bt4DfbMtfC/yY4Yj1IYYjise8nrZ+PHvPL7Cy1du11ZvrNfeF9lw80Pp0XCt7d+vPj1sfPgyE4fV5b9vmV2n/C94mvCYWuwPettMTOyYk2vI7gTe26UuBd7Xpx/xzjlsH8JfA/wCeBuwL/C3wH1vZWQxDW28Bdm3/8B8E1gHPZNhT/RTwnlb/yFb/dxmGPY4HHgb2buUXtjep5cAS4OUMb87Lgftb/V0Y3iTvB5bN9TgwhMSPW9slwHuA6yY8hr2QmHS/fhF4DfDUVvbnwF+OtL22PQeHtcdpt7adTwNLgRUMb8LHtvonAxuB57f6/wX4Uivbh2FI55S2nv/UHtNeSHyHkSAeU75lZ+Hotr63t23v3h7rG4H/2uafA9wGHDPy/H9hZF0r+fnX02hI9J7fx7Rj7tfcPwGvb+t4I3A3PxtGv3b0sQCOafdhKUNgPB/Yb7H/V5/ot0XvgLft9MT2Q+I64Pw2fSlThgTD8ME/0vaY27LTgb9q02cBd46Upb3hHDKy7GXAt9r0kQx70qPbu5fhKGGXVvYrY/p/HvAnWy27CjhzrseBISQ+N1J2KPCjCY/hz4XEXPdrzDpeADwwMn8t8LtjtvOvR+avANa06c8Arx0p24UhTA8CzmAk5FrfNtEPiX+ihU+n/J3AFVtt69vtuXrJ6PPbyt8BfHTk+Z8qJOZ4fv+53ZSvuY0jZU9tbZ898liPhsQrgL/f8hrb3v+DO8rNceOdz3KG4aD5Oohh73Jzki3LdmEY395idHoZwz/tjSP1w7DHt8X9VfXIyPzDwNMZ9pD3ZBjyGtePU5OcOLJsN+Cvprwf39lqe3sm2XWrfkwy8X4leSrDkMaxDENPAHslWVI/OzE8+jj1+vX0Nn0Q8KEk7x8pD8PzuP/ouqqqkoxb9xb3A/tNKN+fYfhny/oebetbzhAw+yd5cKT+EuD/Tlhfz6Tnd9Q0r7l/ftyq6uFW7+mMUVWfT/JhhqOYFUk+AZxbVd/fhvuw0zAkdiJJXszwD/+FbWh+F8Ne3T4T3lBrZPo+hr3Fw6rq2/Pc1n0Mw0KHMJxz2Loff1JVr5/nOhfKXPfrrcDzgJdU1XeSvAD4MsMb+xY1pl3PXcC7q+pPty5Isgo4cGQ+o/NjfI5hKOyjnfK7gV8es75vMzz336qqVfPoe8+k53fUNK+5SX7uca6qC4ALkuzLcMT2NoYjKHV4ddNOIMkvJPl14HKGSwK/Nt91VNVm4LPA+9v6dklySJJ/06n/KPA/gQ+0f0iSLE9yzBTbehS4BPiDJPsnWZLkZUn2YDj5emKSY9ryPZMcmeSA+d6nKe3etrFnkj0Z3uwn3a+9GELkwSTPBH5rxu1fBLwjyWFtW89IcmoruxI4LMm/bVeT/Sbw7Anr+i3g5Unel+TZbX3PTfLxJEsZ3jRPSHJUu4T1rQxv0l9iOBfw/STnJXlKe+wPbzse8zLH8ztab16vuTHuYTh3QruvL07yknbffsjPTrZrAkNix/apJD9g2CM7H/gD4OwZ1ncGw0nLmxmuJvlfTB6+OI/hxOd1Sb7PsCf7vCm3dS7wNeB6huGx9zKMI98FnAT8Z4YTvHcx7A1ur9fyBoY3/S23s5l8vz7IcNL+PobzP/9nlo1X1ScY7vvlbVt/BxzXyu5juIR1LcNQ0iqGq6Z66/omw/mTlcCGJN9juFrpBuAHVXUL8O+BP2z9P5HhMuqftKGyExnOsXyrlf8Rw9VH22Ls8zum3nxfc6M+BJzSPuB3AfALDAH/AMOw2v0MV0ZpAj9MJ0nq8khCktRlSEiSugwJSVKXISFJ6jIkJEldO9SH6fbZZ59auXLlYndDkp5Ubrzxxvuqatm4sh0qJFauXMkNN9yw2N2QpCeVJHf0yhxukiR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSepakJBIcmySW5JsTLJmTHmSXNDKv5rkhfNoe26SSrLPQvRVkjS9mUMiyRKGnwM8juE3g09PcuhW1Y5j+K77VcA5wEemaZvkQIYfZb9z1n5KkuZvIT5MdwTDj5HfBpDkcoYfhbl5pM5JwMdq+PGK65IsTbIfw4+fTGr7AeDtwCcXoJ+SFsnKNVfOq/7ta0/YTj3RfC3EcNNyHvvD5JvasmnqdNsmeTXw7aqa9Bu4kqTtaCGOJDJm2dY/d9erM3Z5kqcy/Nzmq+bceHIOwxAWK1asmKu6JGkeFuJIYhNw4Mj8AcDdU9bpLT8EOBi4Kcntbfn/2/Lj7aOq6uKqWl1Vq5ctG/v9VJKkbbQQIXE9sCrJwUl2B04D1m1VZx1wRrvK6aXA96pqc69tVX2tqvatqpVVtZIhTF5YVd9ZgP5KkqY083BTVT2S5M3AVcAS4JKq2pDkDa38ImA9cDywEXgYOHtS21n7JElaGAvyVeFVtZ4hCEaXXTQyXcCbpm07ps7K2XspSZovP3EtSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKlr18XugLQYVq65cl71b197wnbqifTE5pGEJKnLI4kR7l1K0mN5JCFJ6jIkJEldhoQkqcuQkCR1GRKSpK4FCYkkxya5JcnGJGvGlCfJBa38q0leOFfbJO9L8o1W/xNJli5EXyVJ05s5JJIsAS4EjgMOBU5PcuhW1Y4DVrXbOcBHpmh7NXB4Vf0L4O+Bd8zaV0nS/CzE5ySOADZW1W0ASS4HTgJuHqlzEvCxqirguiRLk+wHrOy1rarPjrS/DjhlAfqqHYSfaZEeHwsx3LQcuGtkflNbNk2dadoC/AbwmXEbT3JOkhuS3PDd7353nl2XJE2yECGRMctqyjpztk1yPvAI8KfjNl5VF1fV6qpavWzZsim6K0ma1kIMN20CDhyZPwC4e8o6u09qm+RM4NeBo9pQlSTpcbQQRxLXA6uSHJxkd+A0YN1WddYBZ7SrnF4KfK+qNk9qm+RY4Dzg1VX18AL0U5I0TzMfSVTVI0neDFwFLAEuqaoNSd7Qyi8C1gPHAxuBh4GzJ7Vtq/4wsAdwdRKA66rqDbP2V5I0vQX5FtiqWs8QBKPLLhqZLuBN07Zty5+7EH2TJG07vyp8gXhJpqQdkV/LIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSuvychKQ5zfdzQOBngXYUhoSkHZbhNjuHmyRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6vJHhyRpO9hRfvDIIwlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktS1ICGR5NgktyTZmGTNmPIkuaCVfzXJC+dqm+SZSa5Ocmv7u/dC9FWSNL2ZQyLJEuBC4DjgUOD0JIduVe04YFW7nQN8ZIq2a4BrqmoVcE2blyQ9jhbiE9dHABur6jaAJJcDJwE3j9Q5CfhYVRVwXZKlSfYDVk5oexJwZGt/GXAtcN4C9FdaNDvKp3C181iI4ablwF0j85vasmnqTGr7rKraDND+7rsAfZUkzUOGnfsZVpCcChxTVa9r8/8BOKKq3jJS50rgPVX1hTZ/DfB24Dm9tkkerKqlI+t4oKp+7rxEknMYhrBYsWLFi+64446Z7s9imHXvcr7tF6vtQm57MS1mvxfruV5MT9bH+8m07SQ3VtXqcWULcSSxCThwZP4A4O4p60xqe08bkqL9vXfcxqvq4qpaXVWrly1bts13QpL08xYiJK4HViU5OMnuwGnAuq3qrAPOaFc5vRT4XhtCmtR2HXBmmz4T+OQC9FWSNA8zn7iuqkeSvBm4ClgCXFJVG5K8oZVfBKwHjgc2Ag8DZ09q21a9FrgiyWuBO4FTZ+2rJGl+FuT3JKpqPUMQjC67aGS6gDdN27Ytvx84aiH6J0naNn7iWpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUtetid0DS9G5fe8Jid0E7GY8kJEldHknsANy7lLS9GBJPAL7JP7n4fGln4nCTJKnLkJAkdRkSkqQuQ0KS1GVISJK6ZgqJJM9McnWSW9vfvTv1jk1yS5KNSdbM1T7J0UluTPK19vcVs/RTkrRtZr0Edg1wTVWtbW/+a4DzRiskWQJcCBwNbAKuT7Kuqm6e0P4+4MSqujvJ4cBVwPIZ+yrpSchLjhfXrMNNJwGXtenLgJPH1DkC2FhVt1XVT4DLW7tu+6r6clXd3ZZvAPZMsseMfZUkzdOsIfGsqtoM0P7uO6bOcuCukflN/OyoYJr2rwG+XFX/OGNfJUnzNOdwU5LPAc8eU3T+lNvImGU1VcPkMOC9wKsm1DkHOAdgxYoVU3ZJkjSNOUOiql7ZK0tyT5L9qmpzkv2Ae8dU2wQcODJ/ALBlKKnbPskBwCeAM6rqmxP6dzFwMcDq1aunCh9JmobnQ2YfbloHnNmmzwQ+OabO9cCqJAcn2R04rbXrtk+yFLgSeEdVfXHGPkqSttGsIbEWODrJrQxXL60FSLJ/kvUAVfUI8GaGK5S+DlxRVRsmtW/1nwu8M8lX2m3c+QpJ0nY00yWwVXU/cNSY5XcDx4/MrwfWz6P9u4B3zdI3SdLs/MS1JKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkrll/vlTaZn4Ns/TE55GEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSuvwW2J2c38QqaRKPJCRJXYaEJKnLkJAkdXlOQpKegJ4o5ws9kpAkdRkSkqQuQ0KS1GVISJK6ZgqJJM9McnWSW9vfvTv1jk1yS5KNSdZM2z7JiiQPJTl3ln5KkrbNrEcSa4BrqmoVcE2bf4wkS4ALgeOAQ4HTkxw6ZfsPAJ+ZsY+SpG00a0icBFzWpi8DTh5T5whgY1XdVlU/AS5v7Sa2T3IycBuwYcY+SpK20awh8ayq2gzQ/u47ps5y4K6R+U1tWbd9kqcB5wG/M2P/JEkzmPPDdEk+Bzx7TNH5U24jY5bVHG1+B/hAVT2UjGs+svLkHOAcgBUrVkzZJUnSNOYMiap6Za8syT1J9quqzUn2A+4dU20TcODI/AHA3W261/4lwClJfg9YCjya5MdV9eEx/bsYuBhg9erVc4WPFtAT5ROhkrafWYeb1gFntukzgU+OqXM9sCrJwUl2B05r7brtq+pXq2plVa0EPgj893EBIUnavmYNibXA0UluBY5u8yTZP8l6gKp6BHgzcBXwdeCKqtowqb0k6Ylhpi/4q6r7gaPGLL8bOH5kfj2wftr2W9X57Vn6KEnadn7iWpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktQ10y/TSXryuH3tCYvdBT0JeSQhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqSlUtdh8WTJLvAncsdj8k6UnmoKpaNq5ghwoJSdLCcrhJktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkOaQ5KdJvjJyW7kN6zg5yaHboXvSduXPl0pz+1FVvWDGdZwMfBq4edoGSXatqkdm3K40E48kpG2Q5EVJ/jrJjUmuSrJfW/76JNcnuSnJXyR5apKXA68G3teORA5Jcm2S1a3NPklub9NnJfnzJJ8CPpvkaUkuaev8cpKTFus+a+dkSEhze8rIUNMnkuwG/CFwSlW9CLgEeHer+7+r6sVV9SvA14HXVtWXgHXA26rqBVX1zTm29zLgzKp6BXA+8PmqejHwawxB87TtcB+lsRxukub2mOGmJIcDhwNXJwFYAmxuxYcneRewFHg6cNU2bO/qqvqHNv0q4NVJzm3zewIrGAJI2u4MCWn+AmyoqpeNKbsUOLmqbkpyFnBkZx2P8LMj+T23KvvhVtt6TVXdss29lWbgcJM0f7cAy5K8DCDJbkkOa2V7AZvbkNS/G2nzg1a2xe3Ai9r0KRO2dRXwlrRDliT/cvbuS9MzJKR5qqqfMLyxvzfJTcBXgJe34ncCfwNcDXxjpNnlwNvayedDgN8H3pjkS8A+Ezb334DdgK8m+bs2Lz1u/KpwSVKXRxKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdf1/tBg9pJWZoOsAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "a = clf.estimators_[0].coef_\n", "b = clf.estimators_[1].coef_\n", "\n", "fig, ax = plt.subplots()\n", "ax.bar(np.arange(a.shape[1]), (a - b).ravel())\n", "ax.set(xticks=[], xlabel=\"Feature\", title=\"Difference in Learned Coefficients\");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That said, the assumption backing this entire process is that the distribution of the data is relatively uniform across partitions. The parameters learned by the each member of the ensemble should be relatively similar, and so will give relatively similar predictions when applied to the same data.\n", "\n", "When you `predict`, the result will have the same chunking pattern as the input array you're predicting for (which need not match the partitioning of the training data)." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:32.978807Z", "iopub.status.busy": "2022-07-27T19:21:32.978335Z", "iopub.status.idle": "2022-07-27T19:21:33.000140Z", "shell.execute_reply": "2022-07-27T19:21:32.999539Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Array Chunk
Bytes 7.63 MiB 781.25 kiB
Shape (1000000,) (100000,)
Count 31 Tasks 10 Chunks
Type int64 numpy.ndarray
\n", " 		\n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " 1000000\n", " 1\n", "\n", "
" ], "text/plain": [ "dask.array<_vote_block, shape=(1000000,), dtype=int64, chunksize=(100000,), chunktype=numpy.ndarray>" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preds = clf.predict(X)\n", "preds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This generates a set of tasks that\n", "\n", "1. Calls `subestimator.predict(chunk)` for each subestimator (10 in our case)\n", "2. Concatenates those predictions together\n", "3. Somehow averages the predictions to a single overall prediction\n", "\n", "We used the default `voting=\"hard\"` strategy, which means we just choose the class that had the higest number of votes. If the first two sub-estimators picked class `0` and the other eight picked class `1` for the first row, the final prediction for that row will be class `1`." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:33.003544Z", "iopub.status.busy": "2022-07-27T19:21:33.003071Z", "iopub.status.idle": "2022-07-27T19:21:33.764805Z", "shell.execute_reply": "2022-07-27T19:21:33.764193Z" } }, "outputs": [ { "data": { "text/plain": [ "array([0, 1, 0, 0, 1, 0, 1, 1, 1, 1])" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preds[:10].compute()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With `voting=\"soft\"` we have access to `predict_proba`, as long as the subestimator has a `predict_proba` method. These subestimators should be well-calibrated for the predictions to be meaningful. See [probability calibration](https://scikit-learn.org/stable/modules/calibration.html#calibration) for more." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:33.768970Z", "iopub.status.busy": "2022-07-27T19:21:33.767713Z", "iopub.status.idle": "2022-07-27T19:21:37.395325Z", "shell.execute_reply": "2022-07-27T19:21:37.392653Z" } }, "outputs": [], "source": [ "subestimator = sklearn.linear_model.LogisticRegression(random_state=0)\n", "clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n", " subestimator,\n", " classes=[0, 1],\n", " voting=\"soft\"\n", ")\n", "clf.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:37.399766Z", "iopub.status.busy": "2022-07-27T19:21:37.399116Z", "iopub.status.idle": "2022-07-27T19:21:37.709516Z", "shell.execute_reply": "2022-07-27T19:21:37.708858Z" } }, "outputs": [ { "data": { "text/plain": [ "array([[8.52979639e-01, 1.47020361e-01],\n", " [1.32021045e-06, 9.99998680e-01],\n", " [8.66086090e-01, 1.33913910e-01],\n", " [8.78266251e-01, 1.21733749e-01],\n", " [1.28333552e-01, 8.71666448e-01]])" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "proba = clf.predict_proba(X)\n", "proba[:5].compute()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The stages here are similar to the `voting=\"hard\"` case. Only now instead of taking the majority vote we average the probabilities predicted by each sub-estimator." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Regression\n", "\n", "Regression is quite similar. The primary difference is that there's no voting; predictions from estimators are always reduced by averaging." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:37.713589Z", "iopub.status.busy": "2022-07-27T19:21:37.712221Z", "iopub.status.idle": "2022-07-27T19:21:37.840937Z", "shell.execute_reply": "2022-07-27T19:21:37.840450Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Array Chunk
Bytes 152.59 MiB 15.26 MiB
Shape (1000000, 20) (100000, 20)
Count 10 Tasks 10 Chunks
Type float64 numpy.ndarray
\n", " 		\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", "\n", " \n", " \n", "\n", " \n", " 20\n", " 1000000\n", "\n", "
" ], "text/plain": [ "dask.array" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X, y = dask_ml.datasets.make_regression(n_samples=1_000_000,\n", " chunks=100_000,\n", " n_features=20)\n", "X" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:37.844677Z", "iopub.status.busy": "2022-07-27T19:21:37.844237Z", "iopub.status.idle": "2022-07-27T19:21:39.712506Z", "shell.execute_reply": "2022-07-27T19:21:39.711753Z" } }, "outputs": [], "source": [ "subestimator = sklearn.linear_model.LinearRegression()\n", "clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n", " subestimator,\n", ")\n", "clf.fit(X, y)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:39.716150Z", "iopub.status.busy": "2022-07-27T19:21:39.715724Z", "iopub.status.idle": "2022-07-27T19:21:40.009651Z", "shell.execute_reply": "2022-07-27T19:21:40.009098Z" } }, "outputs": [ { "data": { "text/plain": [ "array([-149.72693022, 340.45467686, -118.23129541, 327.7418488 ,\n", " -148.9040957 ])" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.predict(X)[:5].compute()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As usual with Dask-ML, scoring is done in parallel (and distributed on a cluster if you're connected to one)." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:40.013029Z", "iopub.status.busy": "2022-07-27T19:21:40.012840Z", "iopub.status.idle": "2022-07-27T19:21:41.818432Z", "shell.execute_reply": "2022-07-27T19:21:41.817786Z" } }, "outputs": [ { "data": { "text/plain": [ "1.0" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The dangers of non-uniformly distributed data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, it must be re-emphasized that your data should be uniformly distributed across partitoins prior to using these ensemble methods. If it's not, then you're better off just sampling rows from each partition and fitting a single classifer to it. By \"uniform\" we don't mean \"from a uniform probabillity distribution\". Just that there shouldn't be a clear per-partition pattern to how the data is distributed.\n", "\n", "Let's demonstrate that with an example. We'll generate a dataset with a clear trend across partitions. This might represent some non-stationary time-series, though it can occur in other contexts as well (e.g. on data partitioned by geography, age, etc.)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:41.821892Z", "iopub.status.busy": "2022-07-27T19:21:41.821356Z", "iopub.status.idle": "2022-07-27T19:21:41.825080Z", "shell.execute_reply": "2022-07-27T19:21:41.824498Z" } }, "outputs": [], "source": [ "import dask.array as da\n", "import dask.delayed\n", "import sklearn.datasets" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:41.827581Z", "iopub.status.busy": "2022-07-27T19:21:41.827321Z", "iopub.status.idle": "2022-07-27T19:21:41.831960Z", "shell.execute_reply": "2022-07-27T19:21:41.831015Z" } }, "outputs": [], "source": [ "def clone_and_shift(X, y, i):\n", " X = X.copy()\n", " X += i + np.random.random(X.shape)\n", " y += 25 * (i + np.random.random(y.shape))\n", " return X, y" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:41.835178Z", "iopub.status.busy": "2022-07-27T19:21:41.834713Z", "iopub.status.idle": "2022-07-27T19:21:41.848108Z", "shell.execute_reply": "2022-07-27T19:21:41.847465Z" } }, "outputs": [], "source": [ "# Make a base dataset that we'll clone and shift\n", "X, y = sklearn.datasets.make_regression(n_features=4, bias=2, random_state=0)\n", "\n", "# Clone and shift 10 times, gradually increasing X and y for each partition\n", "Xs, ys = zip(*[dask.delayed(clone_and_shift, nout=2)(X, y, i) for i in range(10)])\n", "Xs = [da.from_delayed(x, shape=X.shape, dtype=X.dtype) for x in Xs]\n", "ys = [da.from_delayed(y_, shape=y.shape, dtype=y.dtype) for y_ in ys]\n", "X2 = da.concatenate(Xs)\n", "y2 = da.concatenate(ys)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's plot a sample of points, coloring by which partition the data came from." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:41.851628Z", "iopub.status.busy": "2022-07-27T19:21:41.851132Z", "iopub.status.idle": "2022-07-27T19:21:42.114086Z", "shell.execute_reply": "2022-07-27T19:21:42.113586Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAACES0lEQVR4nOydd3gc1dW43zOzTc1WtSX33nvFBhuMaaaZ0JNQ0oAkpH2EL4Hkl4T0RvpHCgmB0ELvYLApxoAL2OAu9y5bsnrdNjP398esZK12JauvZM/7PHqkvTNz75nV7j1zzzn3HFFK4eDg4ODg0F60RAvg4ODg4NC7cRSJg4ODg0OHcBSJg4ODg0OHcBSJg4ODg0OHcBSJg4ODg0OHcBSJg4ODg0OHcBSJQ69DRD4rIssTLUdbEZHPicj73ThejojsFBFf5PVKEflSd43fEURkgYjsbOH4EBGpERG9E8b6UEQmdrSf0xlHkZwmiMgBESkSkZRGbV8SkZXdLEebJlMRGSYiSkRc9W1KqceUUhd0jYQ9AxG5R0Qe7WA3dwEPKqUCnSFTVxL5H4+qf62Uek8pNbbR8QMicl6j44eUUqlKKbMThr8X+Ekn9HPa4iiS0wsX8M1EC9Fb6Yyn3+5CRLzAzUBHlVGX0vgBIYG8BCwSkbxEC9JbcRTJ6cVvgTtFJD3eQRGZLyIfiUhl5Pf8RsdWishPReQDEakWkeUikt3cQJGVx77Iufsj5qjxwN+BeRGzREXk3EtE5BMRqRKRwyJyT6OuVkV+V0Sumdd0VdMRuUXkaREpjFy7qrGJQ0QeEpG/ichrIlIL3BFZ1bkanXOViGxs5j3IEpGXIvf1ITCyyfE/Re63SkQ2iMiCSPtFwPeA6yL3vCnS/nkRyY/cxz4Rua259x+YC1QopY40aR8ZMeVUisiLIpIZ6ftVEfl6E/k2i8gVce6rfpV4q4gcFZFjIvLtRsfniMgaEamIHPs/EfE0Oq5E5HYR2Q3sFpH6//GmyP1eJyLniMiRyPmPAEOAlyPHv9N0pSoiAyLvdZmI7BGRWxqNd4+IPCUiD0feu20iMqv+eGTFtgE4pVe5XYpSyvk5DX6AA8B5wHPAzyJtXwJWRv7OBMqBG7FXLp+OvM6KHF8J7AXGAEmR179qZqwUoAoYG3mdB0yM/P054P0m558DTMZ+sJkCFAFXRI4NAxTganR+Qx8dlRv4ApAGeIE/AhsbHXsIqATOjMjmA7YDSxqd8zzw7WbehyeApyLvxySgoPG9AzcAWRG5vw0UAr7IsXuAR5v0dwm2MhLgbKAOmNHM2LcDrzZpWxmRYVJEpmfrxwCuBdY1OncqUAp44vRd/z/5b6SfyUAxcF7k+EzgjMh9DQPygW81ul4BKyL/u6RGbaOafCaONP38xpHBFXn9LvDXyP9oWkSexY3eywBwMaADvwTWNrmnPwO/T/T3tLf+OCuS048fAl8XkZwm7ZcAu5VSjyilDKXUf4EdwGWNznlQKbVLKeXHniCntTCOBUwSkSSl1DGl1LbmTlRKrVRKbVFKWUqpzdgT1NmtvJ8Oya2U+rdSqlopFcSecKaKSN9G176olPogIlsA+A+2AiDyNH8h8HhTocQ2g10F/FApVauU2hq5tvF9P6qUKo3I/TtsZTa2aV+Nzn9VKbVX2bwLLAcWNHN6OlAdp/0RpdRWpVQt8APg2oisLwKjRWR05LwbgSeVUqHm5AF+HLm3LcCD2EocpdQGpdTayH0dAP5B7P/zl0qpssj/pEOIyGDgLOC7SqmAUmoj8K/IPdTzvlLqNWX7VB7BVpSNqcZ+zxzagaNITjMiE9or2I7YxgwADjZpOwgMbPS6sNHfdUAqgIj8PWJyqBGR70UmqeuALwPHImaTcc3JJCJzReQdESkWkcrIdc2azTpRbl1EfiUie0WkCvuplyZjH27S96PAZSKSiv0U/55S6lgcuXKwn8gbXx8lp4h8O2KqqhTbzNeXFu5bRJaIyNqI+aYC+wm7ufPLsVdaTWkqjxvIjijSp4AbRETDVgqPNCdLM30NiMg5RkReiZgMq4BfxJGz6fvaEQYAZUqpxorzZJ8Bn0T7Z9KAik6U6bTCUSSnJz8CbiH6i3YUGNrkvCHYppAWUUp9WdkRNKlKqV9E2t5QSp2PbdbaAfyz/vQ4XTyO7fAcrJTqi+1HkRbOb0y75QY+AyzFNvn1xTaX0GjsmPGVUgXAGuBT2E+8zU22xYABDG4ilz2A7Q/5LrYyylBKpWOb0eLet9jO82exI4z6R85/rYmsjdmMbc5rSlN5wkBJ5PV/gM8Ci4E6pdSaZvpurq+jkb//hv0/H62U6oPt72kqZ1vTjrd0/lEgU0QaK87WfgbqGQ9saqNMDhEcRXIaopTaAzwJfKNR82vAGBH5jIi4ROQ6YAL26qVNiEh/Eblc7FDjIFAD1IdpFgGDGjtfsZ8Gy5RSARGZgz3B11OMbSYb0cxwHZE7LSJfKZCM/eTcGh4GvoPtG3g+3gkRE8pzwD0ikiwiE7CjqBqPbWDfn0tEfgj0aXS8CBgWWR0AeLBNX8WAISJLaNk5/CGQLiIDm7TfICITRCQZO+T1mYisRBSHBfyOk69GAH4QubeJwOexP1P191YF1ERWol9pRV9FNP8/bvG4UuowsBr4pYj4RGQK8EXgsVaMW6+kZ2L7bRzagaNITl9+gu0oBUApVQpciu30LcWeKC9VSpXEv7xFtEg/R4EybPv4VyPH3ga2AYUiUt/3V4GfiEg1tg/nqUZy1QE/Bz6IRAGd0XigDsr9MLYJpADbib62lff3PPYq6PmIGa85voZtRivEdtw/2OjYG8AyYFdEhgDR5p6nI79LReTjiNnmG9jvTTm2sn2puYEjvo2HiPhzGvFIpL0Q2zH9jSbHH8ZWkK0JG34X2AO8BdyrlKrfJHpnRL5q7JXok/Evj+Ie4D+R//G1cY7/Evh/keN3xjn+aewV5VHs/8+PlFKtVQyXYwedHD3pmQ5xEaWcwlYODm1FRPYCtyml3ky0LM0RCah4D5jeWqe2iNwE3KqUOquFc4YB+wG3UsroDFkTiYisA74Y8R86tIOesBnIwaFXISJXYdvs3060LC2hlCoGmg1yaErE3PVV7DDa0wal1NxEy9DbcUxbDg5tQOyUMn8DbldKWQkWp9MQkQux/S9FxAlndnBoCce05eDg4ODQIZwViYODg4NDhzjlfSTZ2dlq2LBhiRbDwcHBoVexYcOGEqVU0wwYcTnlFcmwYcNYv359osVwcHBw6FWISNOMEc3imLYcHBwcHDqEo0gcHBwcHDqEo0gcHBwcHDpEwhVJJAPrJyLySuR1poisEJHdkd8Zjc69O1K0Zmck7t3BwcHBIcEkXJFgl37Nb/T6LuAtpdRo7Bw+dwFEkt5dD0wELgL+Kr2o9KmDg4NDd1FRUcGxY8cIhVoqJ9N5JDRqS0QGYRcm+jlwR6R5KXZ1NLDTWq/ETre9FHgiUjdhv4jsAeZgp/R2cHBwOO3x+/28/vrrlJWVoWkalmUxa9Yspk5tWserc0n0iuSP2NlaG6ea6F9fKCjyu1+kfSDR2VGPEF1PowGxa0mvF5H1xcXFnS60g4ODQ09k+fLllJSUYJom4XAY0zTZsGEDhw4d6tJxE6ZIRORS4LhSakNrL4nTFje/i1LqfqXULKXUrJycVu2ncXBwcOjVVFdXU1JSQtO0V4ZhsGXLli4dO5GmrTOBy0XkYuy6CH1E5FGgSETylFLHRCQPOB45/wjRFdkGcaIim4ODg0Ono5SiqKiI2tpacnJy6NOnz8kvShCBQABN0zBNM+aY39+qKgLtJmGKRCl1N3A3gIicA9yplLpBRH6LXUnuV5HfL0YueQl4XER+j12jeTR2FTgHBweHTqeuro5XXnmFmpoaRATLshg+fDjnnHMOmpZor0AsGRkZcds1TWPIkCFxj3UWPe/dsBXI+SKyGzg/8hql1Dbs6nDbgdex03jHql4HBweHTuDtt9+msrISwzAa/A0HDhxg+/btiRYtLi6Xi3nz5uFynVgf6LqOz+djypQpXTt2l/beSpRSK7Gjs+pLpy5u5ryfY0d4OTg4OHQZgUCAwsLCuP6Gbdu2MWnSpARJ1jLjxo0jPT2dzZs3U1tby+DBg5k0aRI+n69Lx+0RisTBwcGhJ2EYBiLx4nvsYz2Z3NxccnNzu3XMnmjacnBwcEgoKSkpJCUlxbRrmoZTliIWR5E4ODg4AMFgkB07drBp0yZKS0tZtGgRLperwbHucrlISkpixowZCZa05+GYthwcHE57jh07xrJlywCwLIsNGzYwfPhwrr76avLz86msrGTAgAGMGTMGj8eTYGl7Ho4icXBwOK2xLIvly5dH+T4sy2L//v0MHTqUuXPnJlC6WPbv38+6deuoqqoiJSWFmTNnMm7cuITK5Ji2HBwcTmuKioqwLCum3TAMdu7cmQCJmufgwYO8/fbbVFVVAVBbW8vq1asTHpLsKBIHB4fTmqYhvo0xD1Zgbj/e7PHu5sMPP4zZuW4YBuvXr2/xProax7Tl4ODQYzBNk8OHDxMKhRgwYACpqaldPmb//v3jhvq6LGFkVQrhF3egggau6QO6XJaTUb8SaUowGMQ0zajNiN2Jo0gcHBx6BCUlJbz66qtYloVSCqUUEydOZO7cuc3u6egMdF3nvPPOY/ny5SjDxFQKlxLyQskMDaQBFsZb+9Gn5Z1UDsuy+OSTT9i2bRuhUIh+/foxf/58srOzO0XWPn36UF5eHtPu9XrR9cSVZ5JELoe6g1mzZqn169cnWgwHB4cWUErx2GOPUVdXF9Xucrk477zzujxXFNi5tXb+YwVBFSYvmEy/cBJhsdiTVMkxr5++UwYzccrkhpxWu3fvZsOGDdTW1pKZmcncuXPZvXs3e/bsiTI/uVwurrrqKvr27dthGQ8cOMBbb70V0//cuXOZOHFih/tvjIhsUErNas25zorEwcEh4Rw/fpxwOBzTbhgG+fn53aJIkpOTmeDLQxXbyiwgJq9mHySgmZiiOLprJ7v27mHx4sXU1NSwbt26hkiv4uJili1bhlIqxnFvmiabNm1i4cKFHZZx2LBhLFq0iHXr1lFdXU1ycjIzZ85k/PjxHe67IziKxMHBIeG0lHYknoLpKlznDCf8Qj6ELballOHXDKyINUsphWEYrFy5suHvxpimGdf0pZSipKSk02QcMWIEI0aMQCnVpSa/tuAoEgcHh4TTv3//uFFHLpeLUaNGdZsc+vgcVNDAeGsfh3w1DUqkMYZhNBshFa9dRMjMzOxsUXuMEgEn/NfBwaGLsEyD/Jce4KWvncdzt5zJ2r99j7qy+KG0LpeLhQsXout6wwTpcrnIzs5m9OjRnSaTqgujqgIthsq6puXhvWM+3v7xi1i1tBLweDwxTm9d15k2bVq7Ze4NOCsSBweHdqMqAhgfHMQ6VIlkJOE6cwjaYNupvOa+71Lw0duYoQAAB957mWMb3+OS37+KJyV2kh41ahTZ2dns2LGDQCDA0KFDGTp0aKcUkVLVQULPbUcdqQIRSPXguWIc2pD0uOeLCJMmT+b999+PMmGJCKmpqdTU1MRco+s6Z599NoWFheTn52MYBllZWZx55pmkp8cf51TBidpycHBoF1ZZHaF/boCwCfX+ZbeGa+k4/JkBlt15OWY4GHWN7vEx+bpvMP7Sz3ebnEopQvd9iCr3Q+Ppzq3h/eocpG/8Wh1KKVavXs2OHTvQNA2lFMnJydTW1sYtZ3v22WczduzYhmuVUj2ykmJraUvUVu+9SwcHh4RivLMfQo2UCEDYwli2m/J925E4m+PMUIDi7d37YKcOVqJqQtFKBMBSGB8fa/Y6EeHMM8/k+uuvZ9GiRVx66aUMGzYsbjoVt9uN1+uNurY3K5G24pi2HBwc2oV1oCJ2cgYImqSm5kKcCVdzuekzcHiXy9YYVRWIf8BUqDJ/bLNpsnfvXg4ePEhycjLjx49vqEGSn58f17+ilCIYDMa0ny44isTBwaFdSIoHVRsnNFcp0sdOIi1vGBWHd6PMEz4GTXcz+oJPd6OUIAPSwIqj8dwa2rDoTYKGYfDSSy9RUVHRUCVxx44dLFy4kNGjRzN06FD27t0bE/qrlGLgwIFdeRs9mtNn7eXg4NCpuM4cAu4mU4guaONz0LwuFv2/B8ibciaay43m8pCWN5RzvvdPUnLaPuEqpdi/fz+vvfYaL730Etu3b4/rp4iHlp2CNiYrWlZNkGQP+uTokrQ7d+6kvLy8QVEopTBNk/feew/DMBgyZAj9+/ePymnlcrmYPHlyt+QF66k4KxIHB4d2oU3qh17mx/zgEOgChoU2KhP3pbbD2ZuWwdl3/Z2wvxYzHMSbltHuvQ8ffPABu3btapjgS0pK2L17N5dddlmrfBHuKydgflSA+VEByrDQx2XjWjgM8USH6u7duzeughIRjh8/zoABA1iyZAl79+5l7969uFwuxo0bx6BBg9p1X6cKjiJxcHBoFyKC++xhuM4YhCqtQ/p4kVRvzHnupBTcSSntHqeqqoqdO3dGTfCGYVBaWsrBgwcZPvzkPhfRBNfcQbjmnpjwKysrKdtfRlpaWkNSxeaqHyqlcLvdgF23ffTo0Z26v6W34ygSBweHDiFeFzIg/ua9zuDo0aNxVzKGYXD48GGGDx+OmV+M8cEhVG0IbUQG7oXDmg3rtSyLt956i0OHDjWE9WZmZrJkyRImTJjA0aNHY3wgPp+v0zL4nookzEciIj4R+VBENonINhH5caQ9U0RWiMjuyO+MRtfcLSJ7RGSniFyYKNkdHBy6D5/PF1eRaJpGUlIS4fcPEn4hH3W0GiqDWBsLCd6/HlUdP4pq48aNHDp0CNM0CYfDGIZBSUkJ7733HkOGDGHy5Mnouo7b7cbtdpOcnMySJUt6VEqSnkYiVyRB4FylVI2IuIH3RWQZcCXwllLqVyJyF3AX8F0RmQBcD0wEBgBvisgYpVTrPG4ODg49HqUUqqAK62AlJLvRJ+QwePDguH4QEWHMiJGY/9gKRqNQYwWETIzVhyiZksr69eupqKggIyODWbNmxXXUW5bFgQMHME2T2bNnM3HiRI4dO4bP5yMvL++02hPSHhKmSJQdjF2fZ8Ad+VHAUuCcSPt/gJXAdyPtTyilgsB+EdkDzAHWdJ/UDg49G9MIUbJzIyKQPWY6msudaJFajbIU4ae3Yu0rB9MCXcN4Yw+eG6dy6aWX8vrrrxMKhRrOX7RoEWkBFyFdoGnyYFNxeP8h3jlyoEFp+P1+Xn311WZXFvURWrquk5yczMiRI7vqVk85EuojEREd2ACMAu5TSq0Tkf5KqWMASqljItIvcvpAYG2jy49E2uL1eytwK9AtdQwcHHoChVvW8P4fvoWy7IlTNJ2z7vgTuZPO6MRRirG3svfvxD5tzM2FthIJR1YXkQ2Noae2kvmteXzmM5+hpKQE0zTJyclB13VUZQDM+GmePnQVxKw8TNNs1qGekZHR7DGHlknoek0pZSqlpgGDgDkiMqmF0+M9RsT9BCml7ldKzVJKzcrJyekESR0cejbBqnJW/fZ2wrVVGP5aDH8t4doqVv3mqwSrY0uztp0D2JblpdjW56uBXa26UlkWBRveYc19d/HRv35M6Z4tcc8zPzl2Qok0JmiiCmsQEXJycsjNzW3IsCt9fciQvnb4ceMx3UKVFX9HezgcxufzNfShaVpD9mGH9tEjoraUUhUishK4CCgSkbzIaiQPqM87fQQY3OiyQcDR7pXUwaFncmjNMoiXgFUpDq15vYO7yQPAl4BKTjy7HcBe9L8CNL8RT1kW7//+mxRuWY0RqAPR2P/uC0y6+nYmLP1Sk5PbJ53n6omEX8jH2lsGmoCu4b5oFN5PDsRNW+Lz+bjmmmvIz8+nqKiIjIwMJk6ceFpvKOwoCVMkIpIDhCNKJAk4D/g18BJwM/CryO8XI5e8BDwuIr/HdraPBj7sdsEdHHogodoqTCMU024aIUK1VR3sfSUQInamN4Dl2CuU+Bzb9D6Fmz/ACEZyWikLMxRgy9N/YfjCy0nK6Ndwrj41F6OoJnZV4taQ3OYnefG58Fw/2a414g8jGT5E05imprFhw4aoUF6Xy8W0adPw+XxMnz69NTfv0AoSadrKA94Rkc3AR8AKpdQr2ArkfBHZDZwfeY1SahvwFLAdeB243YnYcnCw6T95Hro7djOg7vaSO3leB3s/jh1k2ZQAUBTVooIGVnEtKmRP3kc+evOEEmmEprso3Lw6WtbpuWhD+p5IZeLSwKPjuWZiq0JvJdmNlpWMRCKspkyZwuTJk3G5XA0/U6dOZdKklizoDu0hkVFbm4GYRwKlVCmwuJlrfg78vItFc3DoZnYBq4Fk7IV528uyZo2awsAZ51Dw8UrMyMTt8iYxYOYiskZN6aB8EwEP0FQhJAOTATviynh9N+bGQtu8ZCn0OQNxeZNB02IzAYvg8iVHN2ka7s9MwTpQgXWwAknxoE/shyS3L/JMRJg9ezbTp08nEAiQlJQUU73QoXNwCls5OCQMhW3NfRnbTFT/XPdLoO2OX2VZHF63nH0rnweBEeNGMrjoKaRsD2SOhsU/h1EXtFPO24BtnFiZeIERwEOATvid/ZhrD0ebpdwaoakpvP7fLzVUSazH5UvhU/e/h8ub1A55HLqDthS2chSJg0PCWAv8L7FP+knYvocOTLK7XoWnrgWj7kSbKwmueQrGXtqODkPAY9iuSgVcQih0Hfv2HcXv95P11nFy6jxI0+DKVA+HJh7gk0d+g6a77DK3wNnfuY9+E+a0794cuoW2KJIeEbXl4HB68hqxSgRAwzLXUVc+G2+qF7evHV/TN74drUQADD8s/3Y7FYkH+Dym8Vn2LH+C/DVvUTwghLjcKAQ9Ffp7kllUMQCtsTLxhxlz4WcYOm8JhVvXoHt85E6Zj8sTPw+WQ+/EUSQODgkjvjXADFus/MsHHFhzDKUUo84ZzoLb5qC722DfL9sdv710tx0m3KLzOh94F1t5XIAdaW/v/F7581so2bOZqtk3oHR3JOJYYWhQ6KljT1IlY/zpDT3VR1t5+2QwdP7FrZf/NKKmpob8/HwqKyvJzc1lzJgxvW5jpKNIHBwSxsXYobXRqxLLDHNg7QCMkB2UuOfdA2DBOd9oQ/RVSn+oiVOPPLV/XCXiLz9O/sv/pmjry6RkVzP+chc54zzAA9jmtyso2rqWsn1bCbtTUK7YCDFTU+xJqjqhSNwa7gtGtV7m05CioiJeffVVLMvCsiwOHTrEpk2b+NSnPkVycvLJO+ghOJnIHBwSxhnYe3B92F9FD2bIxdu/vwgjeCJSyQyZ7Fm1n1Bd7D6RZln4/8DdZCJyJ9vtTagrLeS1O5ey+/VHqThYRsGGMO/83M+B9/zYzvXfAOWU7PwYI+BveTXj0yHdhzYuG8/nZ6AN7tv8uac5SilWrlyJYRhYkag2wzDw+/1s2LAhwdK1DWdF4uCQMAT4PnAV8AGQzAvfDVOyr9HznShGn5PPpEu2oLufB5Zgpyo5ydPq7K/YPpF3f2b/dvlsJTL7qzGnbn32r4T9NahGeanMEGx4MMSQ+S40XQc+wJeeg+71oWpKECOEckWbX1wuF2PnTsM3cWK73o3TjUAgQHV1dUx7fSbiBQsWJECq9uEoEgeHhDMu8gN9cldRuv9QQ7aThV99k1ELd+L21e/OfgA7outhbB9GM4jA/G/DGd+CQAX40jENAytYh9sXXa2wcPNqlNk0fS5YBtQUKfoMANAYMn8Jnzz6GwRI2vIyddOvBAR0Ny63m/79+zN+/Pj2vw2nGS3taWlcE7434Ji2HBx6ELM+MxXd6wKBPrkVjD57RyMlAhAE8yBYK1rXoaYTCMO7v72dpz83i2e/MJc37r6GikMnEi76+mbFvdSywJsq2Nl+z8KTnMa5P3iIlJwBeIKV9PnoMfoUbWHCyKFcdNFFXHzxxU7djjbg8XgYMGBAzK59Xdd7nUJ29pE4OPQwyg5V8NFjm0jJepu5N63A7YvjGzmYBkPePkn0lb1J8dVvX0ZN0aGoVYc7OY3L/vQo3j4Pc3jdCtbcV42EvYzNWcLgvrNRWJR51zHyrnWI64c0TjahlKL66H4QIS1vWK+pHKiUYu/evWzbtg3TNBk1ahQTJkxI6NN/XV0dr776KtXV1YgIlmUxZMgQFi9enHCl7GxIbISjSBw6ROkeWPsHOL4VBp4BZ3yTiK2nG1gD1v+C1iQdumHB2jIY9iQMmttiD0Vb17Hqt1+1M+82Qvd4mXxtEuMvswCTbc8Z5Gz4DinuHHQt4uh3WciANDw3z+o1yqIl3n33Xfbu3duQxFHXdTIzM1m6dGlCJ22lFEVFRVRXV5OTk0N6enrCZGmMsyHRwaEzOLwGHj4fzKDtMDiyFjb8A275ELLHdIMAsyEMuJWdv6oeS8H6IjCXn1SR1BQdIt7DohkKUlUA9VPAuPHTCe/MhHCjvFaGhjrmRx2qRIamd/huEklFRQV79uyJKnRlmibl5eUcOHCAESNGJEw2ESE3N5fc3NyEydBRHIOmg0NzvHwrhGttJQJ2KFOwCpbf2U0CuGDnZVAShJAJQRNqw/DkXrtItS+9met2A98Drid92AfYPo4mPXt1skadUDDWkSEQjrPb3FJYR2Mji3obhYWFcVdVhmFw5MiRBEh0auGsSBwc4hH2Q3F+nAMK9r/dfXKMugX+8FNIM+z06sf99oZ4dzJMvC7OBeuBb2Hv/1BkjdxP1iiLkl1urHAYANF13CnJDDvLhZ1DCySjDNxBCDfZaKhrSHrvT2fi8/niKhJN00hJSYlzhUNbcFYkDg7x0NygNfOc5e3TfXIkZ8L1L0CtDyrc4Oljj3/ts5DaL84Fv8KuE1K/2jA55y434y7Jw9c3C3dKH4addTkX/vJxXL4TZix90iegW0SlbRHAo6ONiR/V1ZsYPHhw3HBbTdMYO3ZsAiQ6tXCc7Q69BiNksuXlfHa9tQ8Exp47ksmXj2tbDqq28MIXYMt/wWzk7HYlw8Lvw8Lvdc2YzWEE4dD7dp6soQsgTooSexWygHimLDvt+wdN2vYAPwV2AIJVdDHhFy5Aldip4iU3FfeVE9AyTo1U7+Xl5bzxxhvU1dUhImiaxrnnnsvgwYNPfvFpiBO11QhHkZwaKEvx0veWU7y3DDOSg0r36PQbncVlPz+/a6KKQrXw5FVwcBXoHjACMPFaWPpvlKb3wEgmC1uRxKtmmAMsa+a6AKAD9gpF1YZApN0FpXoySinKy8sxTZOsrKyEh9j2ZJyoLYdTjoLNhZTuL29QImDnoCreW8bRLUUMnNIFES+eFLjxdTsEuHwv5ExkZdVuHnnzSxT7i8n2ZfPZCTeyeMh59vlK2Y55veUJuHR/Oe//40OKdpbg8roYf8Eo5tw4LWplFQ7UomkudE+8lUdzaMAVwAtEKxMf8JkWrov2gUhK78o8CzREpp1MuYsImZltr0Dp0DKOInHoFRTtLCYciE3jYQQN9m8upN+EfrhdXfR0mTUKskbx7uGV3LfxLwRNe5IuCZTw901/BWDx8X3wzg+h9ridYffcn8GML8Z0VX28hhfvfoOw376XsD/MtmW7qCqq4cK7z6b84E7W/f37VBy0zU15U89i7pd/dmL3eVUB1JVA9rhmzFvfBMqxswp7sJ3plwOf7cx3pMegKgOEX93FkcOH+SS1lCp3mLSMPsyZO5chQ4YkWrzTBkeROPQKUjKTcXldGMFoZRIW+NdHh/jBwRI+M28oXzxnFJrWNSanR/IfblAi9QTNIAdW/hAOb4JwZNNfTSEs+4btrJ92c9T5W17aEbWqAntldfjjo5TsOczKn91A2F/TcOzYpvd568c3c/FP/4M8cx0c+sBe8YjARX+E6Z9vIqUH+AVQChwFhgCnZgZeFTYJPvAxBeFyVqYfxRR7VVJWXs6KFStYtGhRzP6Qw4cP88knn1BTU0O/fv3Izc3Fsiyys7PJy8vrgebK3oGjSBx6BSPOHMqaBzdEWWwUYImwJ82DETJ5bPUB3C6Nzy0cedL+lKUo2llCsDZI7rh+eFNPbs4pqStu+DvJDDOxpoyQpnPFoU22M7wx4Tp4+wcxiqRkXxmWGeuX1F0au5c/jWWGo+U0DepKj1H8t0vo5//E3stS7/x/7WuQMRKGxavvnhX5OXWxthdDyGBDWnGDEqnHNE3WrVsXpUh27tzJBx980LCzvaamhn379qFpGpqmkZWVxSWXXNLrEib2BJx3zKFX4El2c/nPL2DFb9+jpriWkGFR7dZYOawPhm4/RQbCFo9+cICbF4xo8cmy4kglr97zFsEa26lsGRZzbpzGlMtbTpTXL7kfhXWFnFt6hK8c2YohGihFsjLjX1BdENOUPSKToh3FMcrEDJsY/mOYoVhHubJMao4eol+fJjm3wnWw+nfNKJJTH6ukFkIWla74dVqqq6uxLAtN07Asi7Vr1zYokah+IkWlSkpK+Pjjj5kzp3tryRuGwb59+ygoKCA1NZVx48aRlpbWrTJ0lISFLIjIYBF5R0TyRWSbiHwz0p4pIitEZHfkd0aja+4WkT0islNELkyU7A6JIWt4BtfddxnX/d9lvDQhk+fHZ1KeFP0sVBswMK3mIxGVpXj1nrepKakj7DcI14UxQyYfPbqRY9uPtzj+jRNuZkQoyFeObMWrLFIsgxRl0qzKSh8e0zT58nHonuhwZd2jM3jmQHKnzET3xgm1VYr0lGbuqepwizJ3NX6/n127drFnzx5CoTYU3monJSUl7Nu3j4qKCrT+qeDRSLbiPw/7fL6GqKy6urqo9CjxME2T3btjSxRblsW+fftYv349u3fvjquM2ksoFOL555/n/fffZ/fu3WzatImnn36agoLYh5CeTCJXJAbwbaXUxyKSBmwQkRXA54C3lFK/EpG7gLuA74rIBOyKPhOBAcCbIjJGqeYeBx1ORUSEtP6p9B/cl4pjsak7+qf7cOnNPx8d311CsDoYUy7dCJlse20neRPibfKzWTBoIcP1bPTWhMy7k+H8X8c0p/VLZekvL+T9+z+kKL8Yl8/dELVlmQG2Pfd3Aka4IVOv7vaSPXoKmfpu+xvTGN0DIy84uSxdxPbt21mzZg0igoiglGLx4sUMHTq008cKhUIsW7aM0tLShiy5gwYO5KykNKbWZrEu9TimduL/4nK5mD59esNrr9cbN+dYU5qeEwgEeOGFF/D7/YTDYVwuF+vWrWPp0qWdsmrYvHkzVVVVDUqufnX0zjvv8NnPfrbX+GwStiJRSh1TSn0c+bsayAcGAkuB/0RO+w92PCOR9ieUUkGl1H7s3VTduwZ16DF848KxeN3RH1+vW+ObF7a8SzlYG45OgFiPgmD1yZ+oB7lScDXVQgC6D1Jy7ck9ayxc+ShMuCpuH1nDM1j6ywu59YUb+MIT1zHvCzPR3TpuXwoX/vJphp15Ke6UPvjSsxl76ec4++5/wqKfRJfO1dx2rq15d5xU5q6goqKCNWvWYJomhmEQDocxDIM333yTYDDePpa2o5Ti4MGDrFixgqeffprjx483jGWaJkcKCtg+W2fM8NHMqMvBY2loCB63mxkzZjBp0qSGvtxuNyNHjmyxmJSmaTHO+bVr11JTU0M4kl6mvhTuqlWrOuUe9+3bF3elFAqFqKys7JQxuoMe4SMRkWHAdGAd0F8pdQxsZSMi9Y+IA4G1jS47EmlzOA2ZOTyLP904i3+8tZt9x2sYlJnMreeOYu6o7Bav6z82GyNOGLHu1hg2rxU7nMcuha1PQbgmul2AWz+CvoPacBexJKXncMbtv4w9cOaddsjv6nvtqLDRS+DM7zaTJqXr2bNnT0Od8caICAcOHGDMmDGEqitw+ZLbuBfmBKtWrYpK+94U0zTJ37OL2TfdxHQmME0pQqEQbrc77kbDs846C8uy2L9/f8P19f4Tl8tFWloas2ZF77/bv39/zH0qpTh69GiD/6UjNKfYlFItKr2eRsIViYikAs8C31JKVbWwlIt3IO5aVURuBW4FnFjyLsSwDD4sXMexmqMM7TOM6f1noEv3ffinDc3gb19o26K0/HAlogmqiR/FshSjzx528g7GXAJD5tthuOFaQOyVwvxvd1iJnJSxl9o/PQDDMOKaipRSlO7fzot/vI1gVTkIDDvrMmZ94QdtUiglJSUtKpHGctQjIni9zY/hcrk499xzCQaDBAIBPB4PBw4coKqqipycHIYNGxajGLratDRhwgTWrFkTc5/p6em9yuGeUEUiIm5sJfKYUuq5SHORiORFViN5QL0H9AjQ+JFxEHagfAxKqfuB+8FOkdIlwp/mlPpL+c6qO6kJVxMyQ3h0DzlJOfxqwW9J9aQmWrxmyX99N5YZJ626R6d4d9nJd8hrOnzmVdj+NGx90t79PuMWGH5Ol8jbUxk2bBjbt2+PmQCVsjj0zO9R1SUNbQfef4VwoJazvvWHFvusqKjgyJEjuFwuampqTuocFxEGDWq78vZ6vQ0K52QlbUeMGMGuXbuiViX143ZGepVx48ZRWFjIvn37GnxNHo+H888/v8N9dycJUyRiq/oHgHyl1O8bHXoJuBk7jenNwIuN2h8Xkd9jO9tHAx92n8QOjblv418o9ZdgRRIE+g0/R2uO8p9tD3L79K8nWLr4mJbJlsNb8ark2IMihP3h2PZ46C6Y/Gn7pxdgmiabNm1ix44dWJbFiBEjmDlzZotP7yejf//+jBw5MqbiYEbNEYLVpVHnWuEgBevfIVBZ2mx9+LVr17Jt2zaUUg3mpnoHfjx0XcflcjFv3rx230NrmDt3LkVFRdTU1GAYBi6XC4/Hw8KFnRNyLSIsWrSI6dOnU1RUREpKCgMGDOh1OcASuSI5E7gR2CIiGyNt38NWIE+JyBeBQ8A1AEqpbSLyFLAdO37ldidiKzGYlsnHxzc0KJF6DGXwXsGqHqtIXt3/Cv6qOjwkIU0spVbYJG9i/y6WQGEXnaoCxgNdXwdDKcXrr79OYWFhwxP+9u3bOXz4MFdffXW77fAiwsKFCxk9ejR79+5F13VGjx7Nh7/8HME4Fmfd5aautDCuIjl69Cjbt2+PilxqiUGDBpGXl8f48ePx+TqnVkpNTQ2lpaWkpqaSlXVCRq/Xy1VXXcWRI0coLS2lb9++cU1gHSU9Pb3HlNhtDwlTJEqp94nv9wBY3Mw1Pwd+3mVC9UKOlvt58N29fHygjOw0LzcvGMH8MTldOqaK75o66bFE89bWtxhbMjNGiSgU6SP6tGp3e/s5BnwdKMIOljSAbwDxilN1HsXFxRQVFUWZiSzLora2lv379zNq1Kh29y0iDBgwgAEDTtSwzx4zjepjB1FWkzQwluJIVYh1y5aRnJzMxIkTyc62AyN27doV1xdSr+QaT9qLFy/uVL+nUopVq1axZ88edF3HsiwyMjJYsmRJg5LSNI0hQ4Y4/tYWSLiz3aH9HC33c9PfV+MPGpgKCsr9fO+pjXz9grFcNafrPvQuzcXk7ClsKdmMpU48PeqiMz9vfpeN21H0MjeWbqJb0U/hgmAaLT8FdwyFrUQOEV0r5C/YFtoZXTZycXFxXPOQYRgUFhZ2SJHEY8IVt3FozRsYwTo7hU1qDprLTXDW1az/+BMMw0CAPbt2M7/feMaeNb1Z85WmaSxYsKDBBJeXl9fp6Uu2b9/O3r17MU2zQdmWlpaycuVKLrrook4d61SmdxniHKJ4cNXeBiVSTyBscd+KXYS6dGKEr0//Jn3dffBp9pfcpyeRk9yPz0+KzXjbU5g6bgqaFWvKscQib2RXhtHuwl6JNP2fBIEnu3BcSEtLi2uGcblc9OnT+ZUe03KHcMHPn6TPnMupXfAV/DOvpXbGtQQsrWHVoQBTWaw5th3/Pz5kmGTGVRCmaTJkyBAGDx7M4MGDuyQH1tatW2NWQ5ZlceTIkW7ZqX+q4CiSXszH+8uIk/8PgIKyui4bVymF9//+w4/vWM/1jx1iyYpSvnRwKPct+it9vD030+w1s66hakQZph49cbg8LqZ+akIXjlxF/K+aAsq6cFzbn+DxeGLCWEWEMWPGdMmYVkomReljUG4vSnc3a+wUhFLxk7u6guFDhsUoCsuyWLZsGeXl5V0iJ9CsshCRTk2FcqrjKJJeTL8+8R2NhqnI6MLiRDX3/5Pa+/+Ju7qOue8XcfkLBUz648sE/vHPLhuzng92Huf6fz3GRQ98j6sf+RkvbdnY6muT3cl862dfIWdROspjT2+ZI/uy9OcX0HdAV9Zhn0BsfhOwy9+e3YXj2uahpUuX0r9//4YstxkZGVx22WVxHdVKKaziWqyimlalFInH9u3bMY2Tx8EoFB6lIbrGggGTuPjii0lKSopSekVFRbz44ov4/f52yXIyBg8eHHevSFJSEklJp0aJ4e7A8ZH0Ym5aMJz8o5UEwidMJh6XxrxR2aR3pSK576+oJl9s5fdT87d/kHb77THnVwYrqQxWkpeSh/sk1QPrKaw9xsrD71Bn+JmTO4eJWZNYub2IX3/0a1wZ+3BrBkGEf+7+iIO1n+HrZ0Sc1qYBe16HigMwYCYMOsOu3RHB5/NxzdevgK/bk2b35DJKwfaR/B92WVuwlUh/4MouHz01NZXLL7+cYLAGy6ojKSmHeHEuVlENoae2QiQrMh4dz1UT0IamA6BqQhjrC1AFVUj/VFyzByJ9o5WRUorqXcdOHnShIMVy09fwgAaia4gowuFwjAIzTZMdO3ZE5c5SSjWECneE2bNnc/jwYUKhEKZpIiLous7ZZ5/da/Jc9QQcRdKLmTc6h29cOJb7VuxGKYVhKuaPzuaHn5rcpeNazZgarPLyqMnZb/j5w4bfsaFoPS7N/qjdNOFzXDKi5d3Z7xx+m/s++QumMrGUxbL9rzIndy4ffNwPV/Z+tAbTlAIxWHHsMW4OLaGPvxL+fRb4y8EKg+gwYBbc8Dq4Y5++u3eiuB4YAzyBbc46B/gUEGdPSytQSoGlkBYSVJ4gBNyL1/sKtp8mGzsX6lkn+jNMQg9vBH+jlVPIJPT4ZrzfOAMVMAg98DEYlv1zoAJz/VE8N09DyzuxA1sdqmRAuZuCZIlKoliPy7Lfc5/l4tzygQ0RdNrIDCr37Y37PzFNk7Iy2wQYCoVYvXo1e/bsQSlF//79WbBgARkZGTHXtYaUlBSuueYa8vPzOXr0KOnp6UycOLFXh+ImAkeR9HKunD2Ey6YP4kh5HRnJni5didTjGjsWIz8/tn3MmKiJoF6JhK0wYcve7PfQtn+Tm5LLzP6zYq4HqA3Xct8nfyFknbBdB80gHxV+iD8pG68eu2nQsjQ2Hv+EhSt+aJeibby9qGAdvP9LWPTj9t5uJzKDjkZoKUthrDqAue4IBE3ITMJ90Sj0US0VsboHeBdboQAUAt8F/gHYiQ2tnaXEdbgphbG5ELW/AhrnKDMVmCbhV3fh/dLME817ShlRk8Z2Xxm1GA3KxGUJI7IHMdibiSe/gmzDZytBTeG+bhLi0snMzIxrTnO5XOTk5KCU4rXXXqOkpKRhr0lhYSEvvvgi1113XbtNUT6fj+nTp0eteBzahuMjOQVwuzSG56R2ixIBSP/xPUhT+7rPR98f39PwsipY2aBEGhM0gzyz6+lm+95UvBFdi42sCpgBPN4alIp9YhUEnxGEw2ujlQiAEYBP/n3ym+olGCv2YK45bCsRgDI/4ae2YR1uLlNsGXb99qYZeUPAQw2vVE0I4qSOwVBQHcI6UBG3d3WsGtX4Op8Ll65zSelQptRmkhH20C+UxPzaPM4cOZWRl81l4K0LcV84GvclY/DeMR99uL2ayM7Opl+/fjGbJF0uF2PHjqWkpISysrKYDYv1pi+HxOEoEoc24z1zPtlPPYn37IVoubl4Fywg+4n/4lu4oOGcylAlLom/4C31l8RtByLXxFcWg/oMAhWrZNwuYWrWpOa3t1qnRvSNChmYG45BuMmEb1iE3z3QzFVFQDy/lAIONrzShvSNn17fo9s+EnczU4UmUdfpE+zNsG6lMbk2i8tKh3FR2WCGhfrgGm+HWGvpSbhmDkCf3B/xRn9GLrroIiZMmIDX68XlcjFs2DCuvPJKvF4vlZWVzZq+SktLY9odug/HtNWFGAUF1D3xJGZhId4FC0hachHibp2zuafjmTmD7Mcfa/Z4bkpe3IldQ2NS9pRmr5vabxrxkjp7dA+3z7qBJzZ+wKbq11BKEAS3S+OH836AN20AZI+Hok3RF+oemHBNK++q5xKoKsO/7zBJzShLVdpcuPdg4keM6dSbtQC0vDS0UVlYe0pPKCqXhvRLQRudhT5jgG1Oa7w/SRe0yf0bJndjQwHGm/tOHNcEXJqdIPnaSYjv5NNNff6seDm0MjIy4pq+dF1v2CXvkBgcRdJFBFatouwLX0KZJoRC+F98kZq//Y2c555FToOwQrfm5nMTv8C/t/6LoGmbVTQ0ktxJXDf2+mav8+pe7p7zfX6x7mcADTvnLx1xOXm1g7jkyGwWVI2jemQl2VP6MHfAGSS5Iu/nlY/AvxeAGQajDjypkJrXQ/wj7cMMh1j39+9zeN1y3O4ULhh6Dy4t1oSp9U/FDAXZ9+7zHFrzOu7kVEZf8GnyppwJfBrbyV8fMSbYUWOfj+rDfdUEzE2FmBuOgqnQpvbHNWsAogmuc4ahSmqx9paDLrajf2Af3BfZO+PN3aUYy/cSNgyOemox3Yq8cDLJ/bPw3DgVcXXc+JGVlUW/fv1iUr64XC7GjRvX4f4d2o+0N1a8tzBr1iy1fv36bh1TmSaF02ZglTXZbObz0ec7d5J2223dKk8i2VC0nmd3PU1JoJTJ2ZO5dsz19E85kRyxpDrI7sIqctOTGJ5zIv18bbiWdcfWEjD8TO8/k7J3q1nz4AYsw0JZCpfPxaBpeVzw3YVIY5OMvxw2PQJle+zQ3wlXgav9WW4TzfoHfsK+lc9hhmxlPD7nUkZln4dLa3RPuqDfOIm3//ZlKgv2Ygbt0Gzdm8S4y77AgKlnUlv8Ipkj1pGWVwdMw87zNbLN8lhldajjtUhWMlrOiaSTwYc+4WjhMd5JL0Cw15RKYHpNDtO/cgmS3DkrccMw+PDDD9m1axemaTJw4EDmz5/fJbv0T3dEZINSKn5UTNNzHUXS+YS2bqXkyqtRtbUxx9yTJtHvjWXdKk9rqP8ctBQSG87PJ7ThY7T+/fEtOgdpIWVFIGzy0oYjvLm1kGSvzlVzhnDWmJyG/i1L8dtXt/PqJ0fxuDTClsW4vD7c+5kZpCVFTzr+qgCPfeF5zHC0I93lc3HenWcxdHYXF5RKEJZp8MzNszDD0Y7yEZnnMLbfEnx6ZPLUBUtMNhc8w/6Sd5r0IuheHyIalmkwaPZi5n3t12h65xojqv+8mqc92zCahPzqSrh88RJyRg3CNEKU7PgEhSJn7Ax0d/cEhzi0j7YoEse01QWI1wvNpcLupLTXnYVVV0flPT+m7tnnIBTCM3s26b/6Be5G6TOUaVL2la8SeOttEEF0HUlOJue5Z3ANHx7TZ9iwuO2BDzlYUtOwWXLjwQqunjOYr11g11R/7qPDLNt0lJBpEYpE/WwvqORnL2zl15+ODsMs2FSI5hLMJpG/RsBg3weHeq4iUQoOr4EjayFtAIy7Iu5+luYwQ0EsM9a/sa9sJX2TBzMs6yzbZ2Eq2/eUsxR/oITCmi2NhWhYoQAUrH+bXa8/yrhLPtf++4rDsf4mxNleZKLYXXQIM3CE93/3DZSyGvaOnPk/fyRv6pmdKkdzWJbF0aNH8fv95OXlkZrac4uv9UacqK0uwDVqFFpubtSOagBJTib1phsTJFV8Sm+8mbpnnsUMhnh6yhJuHHkt5z20g2//ew2HSu0VVe2jjxF4+x0IBMDvR9XUYJWUUHrrl+P2+ea2Qg6V1EbtuA+ETZ5ad4jjVbad/sl1B6OOA4RNxerdxdQGm+bC0okbySWge3toXWsjBI9eBI9cAG/dDS/fAn8YDMWx+2+aw+VLJjk7L6ZdFw9D+s6JdnwDLs3LuJyLW+zTDAXY/cbjrZahtagxGcSNrhAIBetY9ZuvEq6rxvDXEvbXEPbX8N69XyNQ1bW5xsCuvPj444+zYsUK3n//fZ588klWr17d7hQwDrE4iqQLEBGyHnwALSsLSU21nes+L75LLyHpU1ckWrwGwtvzCW/aBMEgfzn7Czw39WIqkvsSdHtZfbCSL9y/lpLqILWPPAJNcx1ZFsbevRgFBTH9vr+zGH84NteSSxM2HbQfW2sC8UNyRQR/KPraQdNiJ1MA3aMzbnHb7fydiWUaHN34HnvffoaKQ7tOHPjwL3DwPbuuuxmCUA34S+G9a4H3gOqT9i0izP7ij9A9vhMPJZpGcko20oxpKsl98h3e4UDnJvRUSjHAlY6Kt5Pd5SKpupB4kXigOPTBa50qSzzZXn/9derq6giHw4TD4YZ9J/v37++SMS3LIhAInFaKyjFtdRHu0aPJXf8hgXdWYpWU4JkzG3cn137oKOE9e0DXKU7JZO2wGYRdJ2zWCiEYNnlq7UGuDDaTIVXTIM6xrFQPusTfKN032R7jjFHZLN98NOaczBQPWU0KTLm8Li76/tm8/vOVtmwKlKmYcd0U+o1JXNhnzfEC3rrnBkK11ShlgVLkTV/Imd/8HdrHD4DRSPn288ENo8Grg3UXaCZwG00jp5qSN+0sFt/zCNuf/wdVR/eTNWoKE5fegjxWYOcVa4RCUR48hDspFaUszFAwpsAUgO7uvOADZSnCT2zBfaiCaa4sNqaWYooCsZXIkCFDSK3eh2XEZiQww2FCtc1tpOwcysvLqauLVZyGYbB9+3ZGjBjRaWNZlsVHH33UUDLY7XYzd+5cxo4d22lj9FQcRdKFiNtN0gXnJ1qMZnGPGY0yTQ5nDcBtGlGKBGxT09YjFdxwxVKq7/srBKOdvlpGBvrwYTH9fmrWYF7ccCSqWJQAyR4XM4dnAvDlxaNYvbsYf9AgZCo0sRNOfm/ppLgO/wGTc7nxoas5vKGAcNBg0NQ8UrLal6eqs/jgD9/CX3bcViIRjn2yij0rnmBM4x32Atw4GlLdkZVF/ft4H3AcO11J82SNnMSCO/8S1WYs9mAs3xu1OVHcOoO+8RmSKmbj9iVzcM0b7Hwldld/sLKUikO7SB/S8TTy1o5irIMVELaYGMokL5TMnqQqDF0x6pK5DB4xlPL9eWx7/u+YZvSqVvd4yZ3StT4SwzCaDSAJh2OVW0f48MMP2b59e0P6edM0+eCDD/B6vQwbNqxTx+ppOKat0xj3uHF4Z88iN1CBEcdU4tKEkf3SSP3ybbiGD0dSIuGeXi+SnEzGfX+J+yUd3i+V/3fFJJI9OilenSSPTl56Ev/3uVnokVDd/n2TeOJrZ3HjWSOYPiyDy2YM5MHb5jFnZPM5o9w+FyPOHMrYc0cmXInUlR2n4vCuKCUCER/Eiidhyo3gijjWh6SCR4/xmdk8B3zS5vFdMwfivnICMiANUj1oY7LwfHEG7kFZ5E46g6xRUwhWNpNBQNMo27etzWPGw9xeHKXMMg0fc6r7Mb9uAANDKYgImSMmMnjuBbi8J/ZPubxJDJxxDlmjp3aKHM2RlZUV9zOq6zojR3aeWdQ0zSglUo9hGGzYsKHTxumpnHRFIiLDlVL7T9bm0P0Y+/ZjVVbgnjDBjhRrB1n/fgDXL3/F+KJ9bM8eQdjlQXNX4+27H7fm4uKZk9FSUui37FX8y5YRXL0GfdAgUq69Br1//2b7PX9yHgvH9SP/aBXJHp3RuWkxX+iMFA+3nNuzzH2txTJCiMR/DrPCQZh3B+x8EUp22OasZjGBZ4C2JwzUx2ajj41n2rOA46TlDUBze7DC0eZHESElZ2Cbx4uLu5l7E6LSqpzx1V9yZPZi9q98HqUUw8++gsFzzu/yDMz1KeHffvttLMtCKYXL5ULTNDZs2MDWrVuZPHkykybFXwm3lmAw2KxPpKampt399hZaY9p6ltiUpc8AM+Oc69ANGAVHKf38FzD37YXISqLvL35GylVXtbkvSUoi/Sc/5o8hg9+9toOVR18lKfeDSPoRnR98uIpvzPgWCwedTfLSpSQvXdrqvr1unWlD25feu6eTkjMQb59M6kqORh8QF6kD52PpPrQvrYXdr8HRd8G9ithSu/VUdaJk7wE/B6oZea5B/ksmjfNmiqaTlNGPfuNbtT3gpLhm5BHafjw2/5cI2vCMRi+FwXPOZ/Cc9pt6LctqV/2R4cOHc9VVV5Gfn09VVVVUGd1wOMxHH31ERUUFCxYsOElPzePz+XC5XFE77uvJymopM/OpQbP/FREZJyJXAX1F5MpGP58DetZmiNMIpRSln/ksxo4dKH8AVVODqqmh8rt3E9q8ud39Jnlc3HRuH9IHrUU0EzSDsBUkZIX488d/pDLYtU7R3oaIMP/rv8XlTYZIckql3JhmX/ZtGMaK37yHEg3GXgaL7gXtK8305APO6ySpdgJ3AyVAkKQMk3N/kESfgcloLjeay02/iXNZfM/DdqBEI8L+Wna98V/W/e377Hj1IUI1rft/a4P7oi8YaqdN8ej2j1fH8+nJrayVcnJ2797NY489xr/+9S8eeeSRBmd2W0hPT2fevHn07ds35lrDMNi1a1dcp3xr0TSN2bNnx5QLdrlczJkzh1AoRElJCYFAoJkeejctrUjGApcC6cBljdqrgVu6UCaHZgjv2UPF3d/H2LMn5pgKBqn594Nk/vEP7e7//YJVGE3TsGNPmuuOreWCYRe2u++eQQl2DY5V2AWlro38tG8vSs64Gcy743He+sWfwCzDtIYStsYDLo5sPEbh9mLyJvaLnP157JXH49jmLLCVyEhgSbvvKJpHOVFzxCZrlOKS3/sIVD2E7hqOOzl2I15tyTGWf+8awoE6zKAf3eNj23N/5/yf/pc+A2I3nDbFfdZQXFNzsfaX29mCR2Uirs7Z37N3717ee++9Bt+D3+9n3bp1AEycOLHN/RUVFcWkoQfbBFZeXk5ycvt9bxMmTMDn87FhwwZqa2vJzs5m9uzZ7N27l+3bt6NpGpZlMWrUKBYsWNDh6o49iWYViVLqReBFEZmnlFrTFYOLyL+xldVxpdSkSFsm8CQwDDgAXKuUKo8cuxv4IvY38RtKqTe6Qq6eiLF/P8WXXIqqbeapybKwjh3r0Bhhy4hxHgOgwFC9PRV7DXAjdn0OEyjFjprKB37S7l5L95uEzLOwmmwOHDprGxlDHoqMlwd8DfgmcD62tbgCWARcSPw07+3hMPHNZ258fYJA/N3cHz/8KwLV5Q3ZGMxQADMc5KN/3sPiH/2nVSNLmhd9Sm77xG6B9evXN+vAnjBhQpv9Gunp6Rw/fjxuOd+0tLRmrmo9I0aMIDc3l+rqatLT09m5cyf5+fmYptlg9tq7dy8+n4+5c+d2eLyeQmt8JKUi8hbQXyk1SUSmAJcrpX7WCeM/hF3I+uFGbXcBbymlfiUid0Vef1dEJmDXK50IDADeFJExSsV5hD4Fqfrjn1D+gL2JIh4+H97Fizs0xvwBZ/LKvpcasvXWY2Exu/+cDvWdeF7AXhE0/rgEgDeBW4H2pVnxpnrQdIkqeTJy4Q7Ovv1N3L76xgLgx9ib8i4AJrRrrJMzC9u81TSsNQQ0H9Rw7JNVsSl9lKJ4x3os0+j0vFxtoTlHdSAQwLKsmCJYJ2PKlCns3bs3SjlpmkZeXl6HEz8ahsE777zDoUOH0HW9oQZ8PEW4bds25syZc8rUhW/N2uqf2IbXMIBSajP2hN5hlFKrsB/ZGrMUqH8M+g9wRaP2J5RSwUjE2B6gt89urSb00XqI48gDwONB79+PlM98ukNjjM4YzUXDluDVvQiCJjoezcNNE24mJzmnQ323i/0r4ZEL4S9j4cUvQnlHAgU/JrZKINirgfZX1xs+b0hMWO8ZN77fSInUE8B+ZjoZIWA1dlXDtkb7fBrbZNd4cvVhm+/Sm71Kayb5pojebGRad9Hc5J6UlNQu01BGRgYXXnghffr0QdM0NE1jxIgRnH9+x/d7rV69mkOHDmGaJqFQCNM0Y5RIPYZhxDWx9VZa86iRrJT6sInm7Eo7R3+l1DEApdQxEak3Mg8E1jY670ikLQYRuRX7MZMhQ4Z0oajdh2vIEMyDB2MPiJB66y2k3f5VtE5IRPfFybewcNA5rD76AS7RWTDobIb0ScB7uPkxePlWCEdMeWV7YfuzcOt6yIr/dF3qL2HFwRWU+IuZkjOVeQPm49bqzUaDsT/uTT+6FrbpqX14Uz1c/MNzeeMXK7FMiyS9gpScaprmnQr7FYVbDiPa2+ROnhe1p+IEG4H/4YR5ysBekF8W59x4ZAGPAX/H/qr0BT4DXN7iVcMWLGXv209HhQlrLjeD5p4f45TvbubMmcObb74ZU39k9uzZ7X6aHzhwINdddx3BYBCXyxXjIG8Ppmmye/fuuFFb8UhPT2/zaqon05p3sERERhJJliMiVwMdM8a3j3ifmrh2HqXU/cD9YKeR70qhuou0b36d0Pr1qMY5r7xekpZcRN+772pVHyHDYmV+EXsKqxmWk8K5E3PxxdkHMDpjNKMzRneW6G3HMmHZN08oEbBrsYeq4Z0fwtWxSQe3FG/mJ2t/jKVMwlaYVUfe5dldT/Prhffic/mAa7B9E40ViQvbpNWcqUkBr2NHuwfBWAQ1CyFteEPYNUDexH7c9PDVFO0oIW3Dd6AqDH1PZAk4uCbMur+G0HQBvouyLM68448MmNY43NSPXSOkqQ/sV8BkbJdha8gF7mnluTbTPnMH5fu3U37QXpmJCKn9BjHrCz9oUz9dwdChQznvvPNYt24dVVVVpKSkMGvWLEaPtj+fSims/eWoqiDagDS0fq17mBIRfJ2Yids0zRYjyUSk4bjL5eLMM7sn63F30RpFcjv2pDxORAqA/cANXShTkYjkRVYjedg5JMBegQxudN4g4GjM1aco3nnzSP/dvVT+4IdYtbWgLJKvWEr6L37equtLa4J86Z9rqawLUxcySfLo3LdiFw/ccga56T2sYmPVkeg8VfUoCw6+G9NsKYt71/+WoHkitDJgBiioKeClvS9y7djrsD8uf8b2VRRjK4lZwE+g5jgUb4eMEZA+tFHPPwOWY0/ygNoOdb+H+4/BeffCjC80nKnpmh2h9cZb8HYdXDIEPDq1JRbr/hrCDNkp1evNVe//7hss/ds7eFPTIz2838ybYQCvYDvruwaXL5nzfvIYpbs3UXl4N2kDhpMzbmaPsd8PHTqUoUOHxrSrqgChhzai6sKRBGygjcrEffWEbl9Jud1uUlJSqK6OTcaZl5dHcnIyJSUlZGRkMH36dHJyEmAq7kJOqkiUUvuA80QkBdCUUidPW9oxXgJuxn4Uuxl4sVH74yLye2xn+2jgwy6WpUeRvPRyki67FOv4caRPH7Q2hCr+6fUdHK8KYlr2U5E/ZBIMm/z65e384cYetrfUl2GvSuKRGhsZdKT6CH4jNpotZIV498jKiCIBe1/tC9gRWz6wkuCVr8Cmh+10JmYQRpwHVz8JnmLs1Ugjv4pbgywPDAeWfR3S8mB0k9Bdb1/YdMT++9yBHPzA1n8xiHBk3QpGLq6vJ19L/Igrk9ZkCu4oIkL2mGlkj5nW5WN1FqFn81GVgSi7hLW3DOPDAtxnDG7+wi5ARFiwYAHLly9v8IuICC6Xi7POOouMjFNzY249rUmRckeT1wCVwAal1MaODC4i/wXOAbJF5AjwI2wF8pSIfBE4hG2TQCm1TUSeArZjP6bdfrpEbDVGNA09t+1hlqt2HG9QIvVYCtbtLcWyFJrWM54+AfD1gfFXwo7nwWi0gcudDGfFmvE8uruhtns9urLoF/LTJ6WpuUGASFqR1b+BLY/ZCqQ+Um3fm/D6N+HyS4hrTfXqMCINtpXDqp9HKxKl7JTxAJvKYFMZ4ePZWEZWTF/KNJukc59LfEWShP0VcWiMqguhjlbFGrfDFtb6o9DNigRg0KBBXH755WzcuJGKigr69+/P1KlTT4sywK0xbc2K/LwceX0J8BHwZRF5Win1m/YOrpRqLswobhyrUurn2PkfHNpIc2aKtlovQobFu/lFHCiuZVhOCmeP74/H1QVmhMv/ZU/Ku16xa65bJpz9A5h4TcypuSl55Kbkcrj6MArFBSWH+PzRHWgoPKyGykq4/AHwNFnBrftztB8GbMW1+VG45LMQb2e2YUF1JLy28lD0sYPvQXW0tXVgag07yzIxVZMiZyJNfCR5wE3YmwqD2DNkEjAbW8l0LcofRgUNpK+vx5i0WsSwIh/eWL+EMhIXDZWdnc1553VWpoLeQ2sUSRYwQylVAyAiP8L2Pi4ENgDtViQO3cfiif15ffMxjEYFQHRNOGtMTqtXIyXVQb74z7VU+0/4Wf4v4mfJTuu8GheAPelf9wzUFkP1McgcFasIGnH33P/H9977LuPKDvClo/n4GkxjJux4wf7z6v9GX9Rc2hfLAHM66D5QfpDGthMFn5SCaDB4XvR1Rz8iKrEVkJUUYHBaNUfqMjEMWybdm8SoxdfSZ2DTWhhfxo5ofxE7XPhC4Gy6Mkm3CoQJP78Da18Zdi5/HdclY3CN6+E2/DQvpHmgvEnKEV3QJ/Rw2U9BWqNIhhCddyEMDFVK+UUkXmC+A6Asi+DKdwm+9z5aThbJV13VYrbcruYbF45j25FKiioDhAwLj0sjPcXDdy5r/ea4e1/dTkl1gEiJdfwhk5Bh8rvX8vnlddO6RvCUHPvnJAxMHcgDFz6E//6ZjZRIBCOAyn+eXa+tp6JEJ2dkJkPnDkYfuhB2LyPmqTZzFHj6Av8AuQPMY2AE7Updz+6HKsM2s53z4+jr+g4F3XvCvIX90HzG0EqOjb2ZA4UGoumMOPtT9J9Uv8oownbo1wLzsLMAN82R2nWEntyKOlJl35upIGxhPJ+P9jkfWl4aSikqDu1CmQbpw8ahaT0jZFVE8FwxntBjm8G069bj1pBUD64FsY55h66lNYrkcWCtiNQ7vS8D/htxvm/vMsl6MSoUouSzNxDeuAlVVwdeL9W//yOZ/34A38L2ZxjtCH2S3Dz61TNZt7eEfUU1DMlOYf7obFxtSKz3/q7iBiVSj2nB+zuPx7+gm3FpLtL88VcZRgg2PbaK8pp+uH0uUrI38anv/BLPofch7LdXEqLbTvdL/xG5aji1pY+y+9230QveZGjlclLdSWjjzodzfwo546IHGXs5eFLs8roNPhtBXF4GXHY3A3xNbeVvAz/AVmRhbLPWIuyULV1vXrLK/KiC6thSloaFseYwNdPhvXu/Rqi6EkTQPV7O/NYf6D+xe/YBW8W1mFuKIGyhj89GBveNMrtpg/vivX0OxifHUOV+tCHp6JP6Ic2ltnfoMlpUJGL/1x4CXgPOwv50f1kptT5yyme7VLpeSu2TTxH+ZOOJPR/BIAoo++rt5G38GOmEDVDtQdeE+aNzmD+6fUv/5qe2k096gaogG57czP7Vh9A8OhMuHM3ky8ejd7Z/ZchZsPWIve+kEQqNylq7OmM4YFBVWMOHbwQ466tbYc3v4fAa6DcR5t8JOeMBqCqs5tk7lmEEDSxjNmu0OehujQsvPIdBOXE2Mbo88IUP4LnPwrGP7bbs8XDlo3YAQRR+7NiSxov6APaO9vewLcftp66sCDPoJ7X/kGZDYVVVwM7a23SPpgKrtJa3f/o1wrUnUtwbgVre/fWXuezPy0lK79oSx8aHRzDe3Gc/qSgwPz6KNrEf7svGRikTSfPiXjisS2VxODktzmhKKSUiLyilZmL7QxxaQd0zz0RvHKwnHCa8ZQue6W0vYtQTOHtcf97JL4qK/nJpwjnj+7VwFYSDBs/duYza0rqG5IYb/ruZwvxiLvre2W32+NdV+DGDJqn9UmIdw+fcg9r5MipQg6bZY4VND+sKrsVSJz7ulmGx9/2DnHXrHLgofsbkdQ9/QtgfRkXuV1kKI2iy6q/r+PQ/lsZ3SmeOgC+tgbpSe1VSb5bb9Sqs+QPUldip5efPA1+8Cd4PLKO9iqS25Cjv//5bVBzahWganuQ0zvjar8mddEbMuVr/1NjVCIAu1HjKUWZsAgtlWRx4/2XGX9pyrfmOoGqCthJp7DQPW1jbjqOm5CLD0rtsbIf20ZpH47UiMlsp9VGXS3OKIO5msrkqBQlajXQGd1w8jh3HKimrCREMm3jdOpmpHu64eHyL1+1ddQB/ZSAqQ26OZyuzq7+L+vERxJcOZ/wPLPwetGCDrymu5c3fvkfJvjIQIamvj3P/58xGqdqBrFFULV1J8T+/Qf/kfGpDmXxSdDmHqqbF9Hey6KSCTYUNSqQxtaV1BKqDJPVpYWd0cqNiRu/9Elb97ESEWMkOqBwMV+Q1s5hr3ypNWRZv/fhm6kqOoiJ5nPxBP6t+81UuvvdlUvtFZxSSJDf63IGYHxacKEwlgMdFefphrDiKxAoHCVQUt0u+1mLuKYv/voQtzPzjaI4i6XG0ZlZbBNwmIgexPYKCvViZ0qWS9WJSPvMZwps22/6RRkifPrjbUUOhp5CZ6uX+q6ex7L611OwqRfe6mHzxGPp4W/4YHdt2HCNwYlLKTjrAkhH34tYjTulAOXzwK6grhov/HLcPZSle+v4KaoprGyb3muJaXvvJ21x332WkZqc0nOsbOoGVR76C2bRqXyN0t8boRU2jpqLxpHoI1oRiDwi4T3LPDfgr4N2fRO+HMYOwowDMrDjfwCRan1srmuPbPyJYVd6gROpRhsHet55i6qf/J+Ya17kj0HJSMNYcQfnDaCMzcJ89jKzSDOSZWIXm8iXTf2LXhiOLrsVfpQrQFaHmDh2mNf+VJdjVd87F/oRfSns/6acJSVcsxbfkIkhKAo8HSUlB+vQh68EHEp4EryP4KwO89J3Xqc0vtp8Oa0JseSGft+5tLr2HTZ8BaeiN6nfPyH0eXWsyQYfr4ON/QiB+2dmCLYUEqgIxKwTLtNixPLrQlzfVy/D5Q9A9sasb0QWXz0Xm0AxmXd/ys9CUy8bhalJvXXNrDD9jCK7WKpJjH4PuiW0P+uFNwc7Om4SdhdiL/dVq30TtLz9OvH0Vlhmm5nhB3GtEBH1KLt7bZuH71jw8l41D+vjIHD6BgTMXoTdKLql7fGQMn0je1K4NGNFGZ9lh1k3RNfTJnV/zxKHjtCZFykGASBZep8RuKxBNI/PPfyL85S8TXLsWLTMD34UXoiX1sJxWEcI7d1L76GOYx4+TdP75JF1+GeKJnfy2LduFETSiJnMzZHJoQwGVx6rpmxe/MNC480ay6dltDSuEzKQjxN26orntTX6+STGH6kr9cUuxWGGLqqLYdOtnf20euktnz6r9IILLozPq7OGk9Usha3gGA6fkntS0NfHisVQUVLFjxR50t45pWORN6MfCr7Zhok/pR1SxkgYEqodiV2x8BzsH1zzsZ7b2kTVqMipO9lndm0TupLYrp/nfuJcD773EnreeQpkmwxZewcjFV3X5w5D4XLivnkD42e0R+4f9o58zDC234xmuHTofOVntYxG5HPgddn6r48BQIF8p1StsNLNmzVLr168/+Ym9gMLaQt4+9BbVoSpm9p/FjP4z0TpYL6LuhRcpv/NOCIXBNJHkZFwjR5Dz/HNIE8X32o/f5vDHsXkyPcluzvnmfIa3kJbi+K4S3v7jB9Qcr2Xx4D8wtM8GNGny2XP54M6iOBFOUH64kmfveA0zFD1Rurwuzrx1FuPOi59aPhwwCNYESc5IQmtnDXF/RYCyQxWk9UuhT247quj9bRoc3xodSeZOhpvegsGxTvCOsOa+uzi8bjlm0A720FwekrPzWPKb55tJXZ94/BXFBCpKSRswDJfnxLOqCoSxdpaiDAt9dCbSkk/KodMRkQ1KqVmtObc16/OfAmcAbyqlpovIIuwKOg7dyNqja7h3w2+wLAtDGbx16E3GZo7jnnk/QW/nJjHl91Pxne+C/4T9XtXVEd6zl9onnyL1czdHnZ8xpC8FmwtjyspaptXsagTsVN8VOaVkfSeJ9CoPO0o/xeD3t6A1rsToToaZt8VVIgAZg/sydM5ADn1UgBG0J2TNpZGc4WPUgmHNju32uXD7OhbgkJTuY2B6B0wqNyyDJ66Aoi32qgsFS/7c6UoE4Iyv/IKccTPZ/cZjGEE/g8+4kAmXf6lHKpGwv5bVf/o2hVvXoLncKMtiyvXfYtzFNwEgPjf6VMeU1RtozTcsrJQqFRFNRDSl1Dsi8usul8yhgZAZ4g8f/45Qox3TATPAjrJ83j2yknOHtK/EbmjzZohnpvD78b/4UowimXTJWPJf3x2lSHSXRvbITDKHpscdw2/4uWf1D9hXuY+wFW5Irrhy+HRuKdjOCH81kpyFzLsDzvxui/IuvuMsti/bxbbXd2MEDUacOZQZV09svb+iQxjYezxWAxnYhTtbmRgwLQ9uWQdl+8BfCv0mg7trnq5F0xi1+BpGLY7NSdbTWPN/36FwyxosI9RQVGvzE38krf9gBs5clGDpHNpCa76BFSKSCqwCHhOR48QWhXboQnaU5SNx4iGDZpCVh99utyKR5ORmy/dKWuwKI61fKpf+7DxW3beOsoMViCaMOHMoZ315drNjPLTt3+yp2EO4SQ6q/NRM7hh7FgDnD72Ar0//5knl1XSNSZeOY9Kl4056bucSws6DtRt7n4cLeBJ7sd6GCS9zBNBypNjpQqCqjGOb3scyooMuzKCf7S/+y1EkvYzWKJJN2GXb/gd7J3tfwPF4dSMuzY2KXwwSj97+ZInuSZPQsjIx/X4ae7IlOZnUm2+Ke02/0dlc8bslhAMGHq/rpDvTVx5+J0aJNOXtQ2/x2fE3kunLjGqvC9dxqPoQmb5M+iW3vOmxa3kZ2IW98xzs1YmBXYnwLOyIK4e2EKwqR9NdUeV967Gjzxx6E63aR6KUsrCLJfwHQEQ2d6lUDlGMzRyLR/fib1I10Kf7uGDohe3uV0TIeuRhSq69vmHPiwqHSPniF/AtPjfm/OKqAL98aRvr9pailGL6sEy+v3QiAzKaz8prxI1YisaludhfuS9KkTy762n+u/NxXOIibBmMzxzHXXO+T6onEc8wyzihRJqyDZjWfaKcIqTmDo4b/SWaHncXvkPPptnHSRH5iohswS6xu7nRz37AUSTdiC46Pzjjh6S4UkhyJeHRvXg0D4uHnM/s3I4l0HOPHk3u+g/J/Of9pP/m1+Su/oC+d0X7KpRSPLfrWb64/CZ2J99Dn1H/RU8+yicHyvjiP9fhDzWvLMb0nYyyWg6zVShykk7k/1p7dA1P7PwvITNEnVFH2AqxvWw7965PVMWC5hzVCnvvR29lE3Y54R9hl/ptOYKzM9FdHqbfdBd6oygt0V24klKYeNVXuk0Oh86hpRXJ49iPYr8EGpelq1ZKlXWpVKcp2wsqeebDQ5TVBFkwth+XTBuIL7KpbkzGWB5a8ggfFX5IdaiaqTlTGZA68CQ9tg7R9RazEj+8/SFe3PMSuEII4E45TvrIFyjffTWBUB5vbSvi0unxZbGKz0WRj1JhNN1AqehNy5poDO8znCF9TqT+fm7PswQbR3Rhr2y2lGymPFBOhq99ZUuVUuxYvodPnt2GvzJAv9FZnPH5GeSMzDrJlVcBG2mo3d5AH6C7/TWdxT+ARzhRROtt7PxeP6M7Mg8DjFx0FSnZA9j+4r+oKy2k/8Q5TLjiVlKy4yTEdOjRNKtIlFKV2CV1nVDfbuClj4/w+9fyCRoWSsHGg+U8+9EhHrjlDJI89r/Jq3s5a2D3pqEPGAFe3vcSGYezGPPxZJKrU6lJr2THzM0Ec9dRtf8yDpXUNnv9gWNuSstvwpeZjyvpOKKF8aQdRnQDl6Yxrd807ph5Z9Q1FcGKuH3polMdqm63ItnwxGY2Pb+9IXz46JYiXrp7BZ+69yIyh6S3cOXZwFLgOUDHXsi7gT/SXZNu53IU20rd2D/hx46n+RiY2W2S5E6eR+7keSc/0aFH03szCJ5C+EMGv38tn0Cj3FCBsEVBmZ8XNxzh+nnDOnW8yroQmghpSSd3Epf4i8ndN5jx787AZdofl8zj/Zi7/BzWnvMRhkdndAt7SEbkpHKotBZ/8bRGrQpvUh3//coiBvSNVQrT+01n+YHjmE1SweuazoDUAa26x6aEg0aUEqnHDBlseGIz53+npWy7AtyJ/Uz1MXa8yTx6r5N9DfEVYAB4l+5UJA6nBo4i6QHkH61Cj5MzJGhYvL29qNMUyb7jNfz4uc3sPW6nFJkwsC/3XDmFARnNb1bL8GYyat2kBiVSj266mLBhMrXTPZwzrvnKj587ewRr9hRHKUmvW+fC8WNilEipv5QVB5dTE6zGo3kIW2EMZftfvLqXWybfhktrw0e2qsCORus7iJqiGiTOe6wUFO9praV2YOSnDSgFR9ba5YIHzoG+g9p2fZfgI757VAeaD5xwcGgOR5H0AFK9rrg56gD6tmLV0BpqAmFue2AdNQGjwaW69XAFtz6wjie/fgaflHxEcd1xRmeMZWLWxIY8VD7xkVQXX9H0rU7lgVvOwN1CCPDYvD787rMzuPfVfA4U1+Lz6Fw9ezC3LR4ddd720m3cs/qHmMokbIXxal58Lh9Zvmz6p/TnytFXMSGrlVl5ivPh6euhbJf9On04KZc8ErMjv+E+BrQj7UlrqDwCD58H1QV2jXcjCDO+BBf/pc01WDqXs4FfxWnXgYu7WRaHUwFHkfQARuemkZ3m5UhZXVRiQp9b45q5nVN/esWWQsKmFRWXYynwq2K+uPzzKAkTMkO4NTcj00fx4/k/xaN70FwanmQPodrYvSB9s1NJT4mT2bYJM4dn8d+vnYVhWuiaxCRLVErxu/W/JWCeCLENWkEsLObkzeXGCfH3tMQlVAcPLrQLS9XfbUk+nifPY9zCZ9jx3vGofF0ur87M6ya3vv+28NTVULYnOsfWxodg0FyYemPXjNkqUoF7gf/lhInLAL4LDEmUUA69mF6X01xELhKRnSKyR0TuOvkVPR8R4Q83zCSvbxJJHp0Ur47HpfH5hSOZEyeiyDhwgLKvfZ1j02dSdP6F1L3wAidLvllQXhdlXqrHO/A16sxq/IYfU5kEzAC7y3dx/4bH+c+qfTy//gjjLo1Np+7y6sy8tm0TsEvX4mbcLaorpCoUmz4+bIV5v2BVm8Zgx/OR2h9N3g8zzPwz9jDhotG4vDqiC6k5KSy+cwG5J6nw2C4qj0DRppiSv4RrYV38mivdy1xgOfBj7LrxrwOXt+I6BXyE7axfQbTD3uF0pVetSEREB+4DzgeOAB+JyEtKqe2JlazjDMpM5tlvLWDbkUoq/WEmDepL3+TYp32joIDjF12Mqq0Fy8I6fpyKO7+DceAgfb7VfJqRCQP74nNrUcpEXHXovkZP7hFCVojX9y+nfEcebl0DBXcsGk7JygNYpoXu0Zl1/RTGntf+lOeNcWvuhhxcscdOvuKJoqoguohUPeFatJoC5n/xNs743AyMoIk7yXXSVPLtJlQN0szXKxi/5koMe9+E934OFQftWvRn/xCy4mc5bh8+4Jw2nB/AThWzFztLkgd7ZfMA0BN8P+2jsrKS/Px8AoEAgwcPZvjw4Wi9uG5QIuhVigSYA+xRSu0DEJEnsOMye70iAXtlMmlweovnVP/l/+x68I2q4Cm/n+q//B+pt3wJLSUl5hqlFJsOlcesSNx686Z6hYVpgRkZ5y8Vtbz68NWYfgNvmqfdKdnjkZWUzZC0oeyv3IdFI6e87uWi4Uva1tnAOXY6+lCTGiWeVNukhJ2zy5PcxRNF1hhwJ0G4iRy6F8ZfefLrNz4Mr37lRHneyoOw4wW4dT1kj+l0cVvHA9ipYupXIQZ22PAPgAcTJFPH2LdvH++88w6WZaGUYt++fWzZsoXLLrsMXW9fVu3Tkd6mdgcChxu9PkKbw2hOjjIMQps3E96x46Qmo+4mtO5DMGJ3kovbhbF3b9xr3txWyIsbYivkpbrTGZQW+/ZZlk6gfGx0m1JsPVZFUrqvU5VIPd+dczeZSZkkuZLw6l48uodZ/WezZHgbnb/Dzoa8GeBqFCDg8kH2OBh1UecK3RKaDlc8ZKfHr1+ZuJMhbQCc+Z2WrzUNeON/TigRAGXZZrF3ftBlIp+cV4k1ZSkgH2jlKqsHYZom7777LqZpNnzPDcOgrKyMHTt2JFi63kVvW5HEe36OmelF5FbgVoAhQ9rmPAy8+y5lt38dwiGwFFp2NlkPPoB7XM/YwawPGYyxa1dMu6quoeK7d5P+21/jmRRdYfDptYcIhGOz/NYGDD439hv8YfOPMCyDoBlElAcz0Ie6olbVs4nCHzJ49ZMCVu8uIaePj6vnDGZ0bvz6Ik3JTcnlnxf8m43HP6HUX8q4zHFRu91bjQjcuBzW/B4+edCegKfeBPPvtCf37mTMxXDbx/DRX6HiAIy8AKbeDN6T5AurbsY8pyw41HJZ464lvvnRpmc9cLWG48fjJ4c0DIO9e/cycWKvqN3XI+htiuQI0UUgBmFv041CKXU/cD/YFRJb27lRcJSyL95im44imIcOUXLNdeRu+Chu+dnuJu1rtxP6YHWUjPWEN2+m5Mqr6bfiDVxDT0zCNcH4ubB0TchwD+RfFzzIe0dWcbzuOP7qHJ540xVTG11EmNKC2a02YPD5+9dwvCpAIGyhCby++Sj/b+kkzp/cupQXuujM7N92BRaDywsL7rZ/Ek32WFjyp7Zdk5QJVvz0/qQmMn3IhcBTRFeREGA09ibN3oXL5WrW4uB299bNpomht5m2PgJGi8hwEfEA1wMvdVbndU89FbfmtQqFCLz9dmcN0yG8s2eT/offIRnx04SoYJCaf9wf1bZofH88ccxRLl1jeE4qKe4ULhq+hBsn3MRNksXcVBOfSxABr1vD59b55XXTcLVg0npq3UEKKwMNfhhLQTBs8auXtxNuZv+GQzN402DitbZJrjHuZFjwvcTIBMAt2OHB9ZsWk7Dzjf0kYRJ1hOzsbLze2KSbLpeL8ePHJ0Ci3kuvWpEopQwR+RrwBvbuqX8rpbZ1Vv9mURGEYsMZlWliFZd01jAdJvmyy5A+fSm75VaobZLnyjAIbd0a1fTp+UN5Y8sxSqrtiV4XcLk0/t8VkxqUg1FwlJJrr8MqLubrIlzYdyDbz7+afkuWcP6kvJPuF1mZX0QojsJQKHYXVTNhYO97Yk0ol/0DrDDkPw+6B1Cw6KcwoRWO+i4jFXgMO1PwdmAAcAG9dTe8iLBkyRJeeeWVBj+JUorx48czdGjn7N86XehVigRAKfUa8FpX9O1buBD/s8811OZoNCieM+Z2xZDtxj1qVFynOy4XnsnR+ztSfW4e+co8Xtt4lNW7S8jt6+PqOUMY3u+Erb7s1tswDx9uqJg4pmYnY5/8PenTskmee/VJ5UnzxTcFWJYipRNL4daW1nHgwyMADJsziJSs3jmJnRR3Elz9X6grg9oiSB/eZeV524YLO2T4nMSK0UlkZmZyww03cOTIEQKBAHl5eaTFqQ7q0DK9TpF0Jb4Lzsc1bizh/Hzw285OSU7Gd9mluEePPsnVJ+fjA2X86509HCmrY3SqcJN+lDFDc/AtOgdxte1f4Ro4gKQLzse/4k0InHDMitdL6q23xJyf5HFx1ZwhXDUnNvjAKDhq33MTs57y+6n594MkX3NyRXLN3CFsPVIZ5dTXBAZmJjM0OzYkuT3kL9/NB/9cj4idwmrNAxuY96WZTLwoUeGwJzBCJoc/Pko4EGbglFxSMuMruKqiGkK1ITKGpJ+0uiQAyZn2j0OXoWlam4NyHKJxFEkjxOUi5+mnqHn0MfzPPYd4faTceANJVyztcN8rtxfxo+c2E4z4EIorLNZbKfzkgd8wJnQ3Oc89E+Ugbw0Zf/4T+m/vpfaRR1G1tXhmzKDvz3/a5n5UXS2i63HjbqyamjitsSwc14/rzhjCf9ccxK1rWEqRlerl3s/MaJMszVFdXMsH/1wfld4EbGUyZPoA0vonrvpz4Y5ilv34bdtxq8AyLWZ+egrTrzoRPVdTUssbv3iX8sOVaLogmrDwq3MZedawhMl9ulBRUcFHH33EsWPHSEpKYtq0aYwaNarrNqOehkhP2yfR2cyaNUutX78+oTIopbjiD6soqowN6ZxwbCc/XfY73JMm0W/ZqwmQzvYBFU6fiVVaGn3A4yH1tltjKia2RGlNkG1HKslI8TBpUN9O+7JufjGfDx/5BLPJpkrNpTHnxmlMvWJCp4zTVsywySOfe5ZgTbRvzeXVueQn55E7LgelFE99/RUqC6qiouFcXp0rfn0RWcPbV1/F4eRUVVXx7LPPEg6fiDRzuVxMnTqVmTOddPktISIblFKtCqPsbVFbvRJ/yKSkOhj32L7soWBZhHfutJ39gFlYSPVf/0rlz35OYNV7Xb4pUnSdjD/+AUlKgnoTW1IS+oA80r58W5v6SvUpirVV/HP397nz3Tt48+CKZtOftAVlKeK+DUqhzObfn6pQVdw8Xp1FweZCLDP2/oyQyY4VewAo2VNGTXFtTEi1GbbY8oqz8a0r+eSTTzCa+BINw2Djxo1RysWhYzimrW7A69bx6Br+OHsD+vojk5yIHWb8zjuU3XKbHYYcClH7n4fxzJlD1n8ebLMfpS34zl1Ev+VvUPPww5iHD+M9eyHJV1+Nltx6Z7ZhGdz13nc4XH2YkGU/oR/afJBNxRv59qz/7ZB8Q+cM4qPHN8W0i64xdG5snqejNQX8bv1v2V+1H4BhfYZxx8z/ZVBa5+aEsgtlxVl1KQj77YmqrtIfvxaKpahtobqkQ8cpKiqK+yCmaRqVlZVkZ2cnQKpTD2dF0g3omnDVnMH43NFvtzcc5KpPbHOW3q8fWr9+lH31a/Zmw0gYsqqrI/Thh9Q993yXy+kaMZz0e35E1gP/IvWmm9qkRADWHltDQU1BgxIBCJpB1hxbzaGqgx2SLX1gH2ZcMwndoyOa7WPQPTrTrppIxqDo0OKgEeA7q+5kT8UeDMvAsAz2Vuzlrvf+l0C8HeMdYOCU/nFXJC6fq8H/0W9UNlaczAK6V2fwjE7P8OPQiD594mdWsCyL5DZ+vh2ax1Ek3cSXF4/m0umD8LgEnxHCFw5y5cZXOffQeiQlhcz/+wvhjRujkjHWo+rqqHvmme4Xuo1sLt4UVVPkBML2so7n1Zxx7WSuvHcJ06+ZxPSrJ/Kp317ErOunxJy3+thqQmYY1Sh8QKEImSFWH/2gw3I0xpvqZf4XZ0YUnN3m8rnIm9iPYZGVUlK6j8mXj8fVKAxac2skpycx7vzOzObr0JRp06bharKS13WdwYMHO4qkE3FMW51AaPNmwps2ow8aiHfhQiRO1lCXrnHnJeP56nmjKSmvoc/qlSgrE9cld5B87TXoOTkE129odgxVV0fxlVcR3p6PPnAAfb79bZIubmNm3C4mOykbt+YmbEXbnnXRSPemd8oYmUPTyRzacl/FdccJxlFoATPA8bqiTpGjMRMuGkP/cTnsfHMvwdoQw+cNYcjMAVHJLefcOI2c0VlseWkHodoQw84YzJTLx+NJPrVScZi7SjA+LAB/GG1cDq45A5FO3EfUVnJzcznnnHP44IMPCEVW+SNGjGDBggUJk+lUxIna6gAqFKL0818gtO5DlFKIS0dLzyDn+efQB7Q9J5IyTQpnzMIqabKL3uu1N0402nUvSUn0/elPSPn09R29jU6j1F/Cl9+8laB5IrBAEPp6+/LAhQ/h1rpn0txQtJ5ff/jLmNWRT/fxndl3MSt3drfIcboRXrkfc81hqI+sc2lIHy+eW2chnsSmZFdKUVdXh8fjcfJotRInaqubqP7b3wmuXWf7NAIBVE0t5rFjlH396+3qT3SdrH8/gKSlISkp4PUiPh9an7SY1C3K76fqF79ExTGFJYqspGx+cMaPSPem49N9eHUvA1IH8ouzftVtSgRger8ZDEobHDWmW3MzIHUg0/t3zr4Wh2hUXQhzdSMlAmBYqKog5sZjiRMsgoiQkpLiKJEuwlmRdIDCufMwjxyJPeB2k7fxY7T09Hb1a9XWEnj9Dazycrxnzqf4U1ehqqtjT/R4yPt4PVozCRwThaUsDlUdwqO7yUsZkJCNXwEjwFO7nuSdQ28DinMGn8u1Y68jqXGdEodOw9xVQvj5fAjGBhXIiAy8N0xNgFQOHaEtKxLHR9IBVLiZetUiqHh5sFqJlpJC8lUnkvPpAwdixCm0Iy4Xkpq4Hd3NoYnGsL7DotpKqoNsPlxOZoqXKYPT0eKEw3YmPpePmybczE0Tbu7ScRxsJMUTvySJgPSJzbDrcGrhKJIOkHTJpdQ++giEop3LrmHD0DsxPr3PnXdQ/vVvRtcgSUoi5UtfRHr4Ul0pxX0rdvHU2kO4XRpKKfome/i/m2cxsJl8VN0hU224liRXEnp3F7s6RZEBaUiqB1Xuj1YoLg3XbCfE+VTH8ZF0gD53fAt9wEAkJTIh+nxIaioZf/5jp46TtGQJfX/6E7TMTPB4kJRkUm/5En3+985OHacreHfHcZ758DAh06I2aFAXMims9HPn4x8nRJ5VR97lc6/fyI3LPsunX72W/2x7CFM1U0TKodWICJ4bpiLZKeDWwKuDR8d1yRi0PCeb7qmOsyLpAFpGBv3fWoH/5VcIfvQRruHD7VDerKxOHyvl09eTfN21qMpKJDW1x69E6nlmXWyZX6XgWIWfA8U1DMvpPtPcx0Ub+Msnf2qIKjNNg1f2vYRhhfni5NiMyQ5tQ9J9eL8yG6ukFgImkpuKtCbDsUOvx1EkHUR8PpKvubpVqdY7PJamNVsZsadSHYjvK9JEqG2mBHBX8fiOx6JCk8Heeb/swDJumHATXt2x5XcGWieVDXDoPTiKpJcQMize2lbIuj0l9O/rY+nMwQzI6PkRSOdO6M/+4pqY6omaCGNy46ev6CqKmtmMKEB1qBpvkqNIHBzag6NIegF1QYNbHljH0TI//rCJSxeeXHuQX18/nbmjenbSuWvmDmHZ5qMUVvgJhC00AY9L4+7LJ+DuZrPHyL4j+fh4bPYAXfRO23nv4HA64iiSXsBT6w5yuLSu4aneMBWGqfjRs5t59X8XoXdxKG1HSPa6eOi2eby+6Sgf7CqmXx8fV80Zwoh+3R+2/NnxN7KtdGuUecure/n0+M/i0pyvgoNDe3G+Pb2AFVsLY0xDAEHDYt/xGkbn9uyoGJ9b54pZg7li1uCEyjE6YzQ/O/MXPLTtQfZV7iXTl8V1Y6/nnMGLEipXZ1NVsI9PHv0tx/M/wpOcxtiLb2bsxTchmuP4dugaHEXSC0hyx9/rYFkqJjW9Q8uMzRzHLxf8OtFidBm1xQUs//51hAO1oBSGv5bNT/6J6sKDzP7SjxItnsMpijML9QKumjMEXxNlIgIDM5MYnOVEyDicIP/lBzFCARqXkzRDAfatfI5AVVkCJXM4lXEUSS/gwsl5XDA5F49LI8mjk+zRyU718uvrpydatE7DtBT7i2uaLUns0DpKd29CmbFh1brbS1XBvgRI5HA6kBDTlohcA9wDjAfmKKXWNzp2N/BFwAS+oZR6I9I+E3gISAJeA76penDGScvvJ7B8hZ14cf483GPGtLsvTRO+t3QSN541nM2HK8hK9TJ7RFaPdrK3hXe2F/Krl7YTMi1MSzFxYF9+fu1UMlOdcNy20mfQSMr3b0epaJ+aGQ6RkuOkKnHoGhK1ItkKXAmsatwoIhOA64GJwEXAX0Wk3qbzN+BWYHTk56Juk7aNhDZvpnDmLCr+9ztU/vRnHF9yCeV33Bm3dnRbGJyVwiXTBnLGqOxOVSLGgQOUfOYGCoYMo2DkaMq//b9Y8bINdwG7jlVxz3NbqPSH8YdMQobFlsMV/M+jzRf5cmie8Zd9Ac3jiWrT3V7ypp5JSnbba+Q4OLSGhCgSpVS+UmpnnENLgSeUUkGl1H5gDzBHRPKAPkqpNZFVyMPAFd0ncetRlkXp57+AqqxC1dZCIACBAP6XX8b/8iuJFi8Gq6KC4ksvJ/jee2CaEAhQ99xzlFz/mQ4rvtbw1NqDhJtEpBmW4mBJLbsLu0eZnUqkDxnD2d/5O2l5wxBNR3N7GLrgMuZ/495Ei+ZwCtPTorYGAmsbvT4SaQtH/m7aHhcRuRV79cKQIUM6X8oWCG/ZgqquiWlXdXXUPf44yZdf1q3ynIzap55GBQLRteJDIYxduwht+BjvrJldOv7RigBWHH2laxqlNUFG07NDm3si/SfN5dI/LiPsr0V3e9BcvSMvm0PvpctWJCLypohsjfOztKXL4rSpFtrjopS6Xyk1Syk1Kycnp62idwgVCtshVfGOBZupX5JAwtu2Raenb4SxZ3eXjz9nZCbeODvcw6bF2LzuTaFy6mB/NdxJKY4ScegWukyRKKXOU0pNivPzYguXHQEa71obBByNtA+K097j8EybCnrsvg9JSiL56qsSIFHLuCdNQpLi5+xyj259gIBZVER49+42F/S6cvYQ+iS5ceknlK/PrXPt3CFkpHhauNIhGgU8A1wIzAYuB95MqEQOpw89Lfz3JeB6EfGKyHBsp/qHSqljQLWInCF23dabgJYUUsIQt5vM+/5iT84Rp6ckJ+OePq1bMgS3lZRrrkZ8Pmi869njwTV2LO4ZJw8vNktLKb76WgrPmE/xxZdybNoM6l59rdXj90ly8/BX5nPt3CEMykxiwsC+fH/pRG4/v/1RbqcnTwN/BEojr49iB0auTIw4DqcVCanZLiKfAv4C5AAVwEal1IWRY98HvgAYwLeUUssi7bM4Ef67DPh6a8J/u7Jme0uYR49R+8wzWCUl+BYuxHvuoh6bosI4eJCK7/0/gu+/j7jdJH3qCvr+6IdorSjje/ziSwlv2waNViKSlET288/imTy5K8V2aEAB52N/lZoyAniqW6VxODVoS832hCiS7iRRiuR0ILxzJ8WXXBbrY9E0kj71KTI7uVKkQ3MEgIVAbD428AIfdK84DqcEbVEkPfMR2aFXYBYdB1ecwD/LwjxyuPsFOm3xAs0FJgxqpt3BofNwFIlDu3FPmogKxYlE83rxLlzQ/QKdtgjwZcDXpN0LfL37xXE47XAUiUO70TMzSb31luioL7cbrW9fUm++OXGCnZZcDXwH6I/9tR4C/Bw4K5FCOZwm9LQNiQ69jD7f/Q7u8eOpuf+fWBXl+M4/n7Sv3Y7Wy2rLnxpcHvlxcOheHEXi0CFEhOSll5O81JnAHBxOVxxF4uDQFYRqYcO/IP8ZSMqCuV+HEYsTLZWDQ5fgKBKHXo1lWuxfe5h97x/E5XMx/oJR5I7vl1ihQnXwz7lQvg+MSGj0vhWw8Aew4K7Eyubg0AU4isSh12KZFst++g6F+cUYAQME9n1wkOnXTGLGNQncDLnxIajYf0KJAITr4N0fw8xbIDkrYaI5OHQFTtSWQ6/l0PqCE0oEQIERNPn4yS3UltUlTrCdL9uKoym6B46sjW13cOjlOIrEodeyf+3hE0qkEaJrFGwuTIBEEVL7gcT5aikLkjK7Xx4Hhy7GUSQOvRZviifufC0ieJISmD599u3garo5UCA5BwadkRCRHBy6EkeROPRaxp43Es0VJ2W/BoOmD0iARBEGzYElfwZ3Cnj72L8zR8FNK5qtVePg0JtxnO0OvZasYRnM/9IsVv9rPZpuPxNpurDkh4tweWIVTLcy44sw6Xoo+Ah8fSF3mqNEHE5ZHEXi0KuZcOFoRp45lKNbCnH5XAyYnIsep+JiQvCkwPBzEi2Fg0OX4ygSh16PN9XD8HlDEi2Gg8NpSw95dHNwcHBw6K04isTBwcHBoUM4isTBwcHBoUM4isTBwcHBoUM4isTBwcHBoUM4UVsOCaEqWMnL+17m46IN5CTncMWoTzEuc3yixXJwcGgHjiJx6HYqghV88+2vUROuIWyF2VOxmw1F6/nq1K+xaMi5iRbPwcGhjSTEtCUivxWRHSKyWUSeF5H0RsfuFpE9IrJTRC5s1D5TRLZEjv1ZxNkm3Ft5dtfTVIeqCVthABSKoBnkH5v/1tDm4ODQe0iUj2QFMEkpNQXYBdwNICITgOuBicBFwF9FpD7Xxd+AW4HRkZ+Lultoh1gM06KyLoRlqVZfs6FoPYaKzdqrUBRUF3SmeA4ODt1AQkxbSqnljV6uBa6O/L0UeEIpFQT2i8geYI6IHAD6KKXWAIjIw8AVwLJuE9ohCstSPPjuXh5bfYCwaZHsdfGVxaO5Ytbgk17bx9MXOBLTblgGaZ7ULpDWwcGhK+kJUVtf4IRCGAgcbnTsSKRtINEzT327Q4J46L19PPLBfupCJmFTUVkX5o+v72DF1mMnvfaK0Z/Cq3uj2lziYmzGOLKSsrtKZAcHhy6iyxSJiLwpIlvj/CxtdM73AQN4rL4pTleqhfbmxr5VRNaLyPri4uKO3IZDHCxL8dgHBwiEraj2QNjiX+/sOen1Z+TN45ox1+HRPCS7kvHoHkamj+KuOXd3lcgODg5dSJeZtpRS57V0XERuBi4FFiul6pXCEaCxbWQQcDTSPihOe3Nj3w/cDzBr1qzWG+8dWoU/bBIIm3GPHa8MtqqPa8dexyUjLmV/5X4yfRkMSHUWmA4OvZVERW1dBHwXuFwp1bi49UvA9SLiFZHh2E71D5VSx4BqETkjEq11E/BitwvuAECyR6dvcvwKhMP7td7HkeJOYVL2JEeJODj0chLlI/k/IA1YISIbReTvAEqpbcBTwHbgdeB2pVT9o+9XgH8Be4C9OI72hCEifO38sfjc0R8fr1vjaxeMSZBUDg4OiUJOWJVOTWbNmqXWr1+faDFOSVbtOM4/3t5NYYWfEf1Suf38sUwbmpFosRwcHDoBEdmglJrVmnOdne0O7WbhuH4sHNcv0WI4ODgkGEeR9FKUaRJ4800CK99Fz8oi+bprcQ0++R4OBwcHh87GUSS9EBUKUfLpzxDevAVVVwduNzV/+zsZf7uPpAsuSLR4Dg4Opxk9YUOiQxupffoZwps220oEIBxGBQKUf+NbqFAoscI5ODicdjiKpBfif+55lN8f54gi9Mkn3S6Pg4PD6Y2jSHoh4vPGP2ApxNvMMQcHB4cuwlEkvZCUGz6LJCfHtEtaKu4pUxIgkYODw+mMo0h6Ib6LLiL56qvA60WSkpDUVKRvX7L+8xCiOf9SBweH7sWJ2uqFiAjpv/wFqbfcQnDNGrSMDHznLkJ8vkSL5uDgcBriKJJejGvEcFwjhidaDAcHh9Mcxw7i4ODg4NAhHEXi4ODg4NAhHEXi4ODg4NAhHEXi4ODg4NAhHEXi4ODg4NAhTvl6JCJSDBxMtBxtJBsoSbQQHcCRP7E48ieWU0X+oUqpnNZccMorkt6IiKxvbUGZnogjf2Jx5E8sp6P8jmnLwcHBwaFDOIrEwcHBwaFDOIqkZ3J/ogXoII78icWRP7GcdvI7PhIHBwcHhw7hrEgcHBwcHDqEo0gcHBwcHDqEo0h6GCJykYjsFJE9InJXouVpCyIyWETeEZF8EdkmIt9MtExtRUR0EflERF5JtCztQUTSReQZEdkR+T/MS7RMbUFE/ify2dkqIv8VkR5dG0FE/i0ix0Vka6O2TBFZISK7I78zEiljSzQj/28jn5/NIvK8iKSfrB9HkfQgREQH7gOWABOAT4vIhMRK1SYM4NtKqfHAGcDtvUx+gG8C+YkWogP8CXhdKTUOmEovuhcRGQh8A5illJoE6MD1iZXqpDwEXNSk7S7gLaXUaOCtyOueykPEyr8CmKSUmgLsAu4+WSeOIulZzAH2KKX2KaVCwBPA0gTL1GqUUseUUh9H/q7GnsQGJlaq1iMig4BLgH8lWpb2ICJ9gIXAAwBKqZBSqiKhQrUdF5AkIi4gGTiaYHlaRCm1Cihr0rwU+E/k7/8AV3SnTG0hnvxKqeVKKSPyci0w6GT9OIqkZzEQONzo9RF60UTcGBEZBkwH1iVYlLbwR+A7gJVgOdrLCKAYeDBinvuXiKQkWqjWopQqAO4FDgHHgEql1PLEStUu+iuljoH9cAX0S7A8HeELwLKTneQokp6FxGnrdfHZIpIKPAt8SylVlWh5WoOIXAocV0ptSLQsHcAFzAD+ppSaDtTSs80qUUR8CUuB4cAAIEVEbkisVKcvIvJ9bHP1Yyc711EkPYsjwOBGrwfRw5f2TRERN7YSeUwp9Vyi5WkDZwKXi8gBbJPiuSLyaGJFajNHgCNKqfpV4DPYiqW3cB6wXylVrJQKA88B8xMsU3soEpE8gMjv4wmWp82IyM3ApcBnVSs2GzqKpGfxETBaRIaLiAfb0fhSgmVqNSIi2Pb5fKXU7xMtT1tQSt2tlBqklBqG/b6/rZTqVU/DSqlC4LCIjI00LQa2J1CktnIIOENEkiOfpcX0omCBRrwE3Bz5+2bgxQTK0mZE5CLgu8DlSqm61lzjKJIeRMTB9TXgDewv0FNKqW2JlapNnAnciP00vzHyc3GihTrN+P/t3T9IVWEcxvHvAwpFChW2RYtDQw7G1aGmCrQthNwaFKShwS2naOkfRWMNTeESBVkNtdhdbLHBDLWCRIIGwc2Gipbi13Dei7ebHrue/MvzAcHj+fO+i/54X899foPAA0kzQDtwY3On8+/SSmoEeAu8I/v7tKXjRiQ9BF4DhyXNSxoAbgJdkuaArnS8Ja0w/7tAM1BOv8P3Vn2OI1LMzKwIr0jMzKwQFxIzMyvEhcTMzApxITEzs0JcSMzMrBAXErMckn5Vvco8laJf6n1Gz3qGV0rqS0mzc+mDZGYbyq//muWQ9C0imgo+Yxh4EREjddzTUBWcl3fdfuAN0EEWpzMJlCLiyxqna1Y3r0jM6iSpJOmVpElJo1VxGOclTUialvQkfUL7OHAGuJ1WNK2SxiR1pHtaUiwLkvolPZb0HHgpaU/qFzGRQhiXS4I+DZQjYjEVjzJ/x4KbrSsXErN8u6u2tZ6lLLE7QG9ElID7wPV07dOI6IyISh+QgYgYJ4vMGIqI9oj4tMp4x4C+iDgFXCKLaukETpIVo9o03x2TGG3bV8NmT8Bsi/sREe2VA0ltQBtZfARkzZcW0uk2SdeAvUATWdRNvcoRUekP0U0WJHkxHe8CDvFn/tSOSIy27c2FxKw+Aj5ExHItbIeBnoiYltQPnFjhGT9Z2g2obSX7vWassxExmzOf+ZpxDgJjOdeb/Xfe2jKrzyxwoNILXVKjpCPpXDOwkLa/zlXd8zWdq/gMlNL3vTljjQKDKQkXSUdXuKZb0r7Uz6Obta2EzNbMhcSsDqkFci9wS9I0MMVSz4zLZB0hy8DHqtseAUPpH+atZF0AL0gaB1pyhrsKNAIzkt6n49r5LKafT6SvK1VbY2Ybwq//mplZIV6RmJlZIS4kZmZWiAuJmZkV4kJiZmaFuJCYmVkhLiRmZlaIC4mZmRXyG6u2TJLhMHBrAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "fig, ax = plt.subplots()\n", "ax.scatter(X2[::5, 0], y2[::5], c=np.arange(0, len(X2), 5) // 100, cmap=\"Set1\",\n", " label=\"Partition\")\n", "ax.set(xlabel=\"Feature 0\", ylabel=\"target\", title=\"Non-stationary data (by partition)\");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's fit two estimators:\n", "\n", "1. One `BlockwiseVotingRegressor` on the entire dataset (which fits a `LinearRegression` on each partition)\n", "2. One `LinearRegression` on a sample from the entire dataset" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:42.117580Z", "iopub.status.busy": "2022-07-27T19:21:42.117135Z", "iopub.status.idle": "2022-07-27T19:21:42.274268Z", "shell.execute_reply": "2022-07-27T19:21:42.273258Z" } }, "outputs": [], "source": [ "subestimator = sklearn.linear_model.LinearRegression()\n", "clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n", " subestimator,\n", ")\n", "clf.fit(X2, y2)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:42.278251Z", "iopub.status.busy": "2022-07-27T19:21:42.277494Z", "iopub.status.idle": "2022-07-27T19:21:42.372413Z", "shell.execute_reply": "2022-07-27T19:21:42.371761Z" } }, "outputs": [ { "data": { "text/plain": [ "LinearRegression()" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "X_sampled, y_sampled = dask.compute(X2[::10], y2[::10])\n", "\n", "subestimator.fit(X_sampled, y_sampled)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Comparing the scores, we find that the sampled dataset performs much better, despite training on less data." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:42.377059Z", "iopub.status.busy": "2022-07-27T19:21:42.376867Z", "iopub.status.idle": "2022-07-27T19:21:42.657861Z", "shell.execute_reply": "2022-07-27T19:21:42.656786Z" } }, "outputs": [ { "data": { "text/plain": [ "-11.8766583738708" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf.score(X2, y2)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "execution": { "iopub.execute_input": "2022-07-27T19:21:42.662146Z", "iopub.status.busy": "2022-07-27T19:21:42.661697Z", "iopub.status.idle": "2022-07-27T19:21:42.794595Z", "shell.execute_reply": "2022-07-27T19:21:42.793958Z" } }, "outputs": [ { "data": { "text/plain": [ "0.09217254441717304" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "subestimator.score(X2, y2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This shows that ensuring your needs to be relatively uniform across partitions. Even including the standard controls to normalize whatever underlying force is generating the non-stationary data (e.g. a time trend compontent or differencing timeseries data, dummy variables for geographic regions, etc) is not sufficient when your dataset is partioned by the non-uniform variable. You would still need to either shuffle your data prior to fitting, or just sample and fit the sub-estimator on the sub-sample that fits in memory." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 4 }