{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Blockwise Ensemble Methods\n",
"\n",
"Dask-ML provides some [ensemble methods](https://ml.dask.org/modules/api.html#module-dask_ml.ensemble) that are tailored to `dask.array`'s and `dask.dataframe`'s blocked structure. The basic idea is to fit a copy of some sub-estimator to each block (or partition) of the dask Array or DataFrame. Becuase each block fits in memory, the sub-estimator only needs to handle in-memory data structures like a NumPy array or pandas DataFrame. It also will be relatively fast, since each block fits in memory and we won't need to move large amounts of data between workers on a cluster. We end up with an ensemble of models: one per block in the training dataset.\n",
"\n",
"At prediction time, we combine the results from all the models in the ensemble. For regression problems, this means averaging the predictions from each sub-estimator. For classification problems, each sub-estimator votes and the results are combined. See https://scikit-learn.org/stable/modules/ensemble.html#voting-classifier for details on how they can be combeind. See https://scikit-learn.org/stable/modules/ensemble.html for a general overview of why averaging ensemble methods can be useful.\n",
"\n",
"It's crucially important that the distribution of values in your dataset be relatively uniform across partitions. Otherwise the parameters learned on any given partition of the data will be poor for the dataset as a whole. This will be shown in detail later."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's randomly generate an example dataset. In practice, you would load the data from storage. We'll create a `dask.array` with 10 blocks."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:33.459837Z",
"iopub.status.busy": "2021-10-14T16:48:33.459057Z",
"iopub.status.idle": "2021-10-14T16:48:38.188405Z",
"shell.execute_reply": "2021-10-14T16:48:38.188810Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.\n",
"Perhaps you already have a cluster running?\n",
"Hosting the HTTP server on port 38395 instead\n",
" warnings.warn(\n"
]
},
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
160.00 MB
16.00 MB
\n",
"
Shape
(1000000, 20)
(100000, 20)
\n",
"
Count
10 Tasks
10 Chunks
\n",
"
Type
float64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from distributed import Client\n",
"import dask_ml.datasets\n",
"import dask_ml.ensemble\n",
"\n",
"client = Client(n_workers=4, threads_per_worker=1)\n",
"\n",
"X, y = dask_ml.datasets.make_classification(n_samples=1_000_000,\n",
" n_informative=10,\n",
" shift=2, scale=2,\n",
" chunks=100_000)\n",
"X"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Classification\n",
"\n",
"The `sub-estimator` should be an instantiated scikit-learn-API compatible estimator (anything that implements the `fit` / `predict` API, including pipelines). It only needs to handle in-memory datasets. We'll use `sklearn.linear_model.RidgeClassifier`.\n",
"\n",
"To get the output shapes right, we require that you provide the `classes` for classification problems, either when creating the estimator or in `.fit` if the sub-estimator also requires the classes."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:38.195685Z",
"iopub.status.busy": "2021-10-14T16:48:38.193576Z",
"iopub.status.idle": "2021-10-14T16:48:38.203784Z",
"shell.execute_reply": "2021-10-14T16:48:38.204369Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"BlockwiseVotingClassifier(classes=[0, 1],\n",
" estimator=RidgeClassifier(random_state=0))"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sklearn.linear_model\n",
"\n",
"subestimator = sklearn.linear_model.RidgeClassifier(random_state=0)\n",
"clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n",
" subestimator,\n",
" classes=[0, 1]\n",
")\n",
"clf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can train normally. This will *independently* fit a clone of `subestimator` on each partition of `X` and `y`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:38.212226Z",
"iopub.status.busy": "2021-10-14T16:48:38.211683Z",
"iopub.status.idle": "2021-10-14T16:48:42.893804Z",
"shell.execute_reply": "2021-10-14T16:48:42.894629Z"
}
},
"outputs": [],
"source": [
"clf.fit(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of the fitted estimators are available at `.estimators_`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:42.901868Z",
"iopub.status.busy": "2021-10-14T16:48:42.901284Z",
"iopub.status.idle": "2021-10-14T16:48:42.929918Z",
"shell.execute_reply": "2021-10-14T16:48:42.930729Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0)]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.estimators_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These are different estimators! They've been trained on separate batches of data and have learned different parameters. We can plot the difference in the learned `coef_` of the first two models to visualize this."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:42.934566Z",
"iopub.status.busy": "2021-10-14T16:48:42.933254Z",
"iopub.status.idle": "2021-10-14T16:48:43.206023Z",
"shell.execute_reply": "2021-10-14T16:48:43.206998Z"
}
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:43.210599Z",
"iopub.status.busy": "2021-10-14T16:48:43.209550Z",
"iopub.status.idle": "2021-10-14T16:48:43.518474Z",
"shell.execute_reply": "2021-10-14T16:48:43.519309Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEFCAYAAAAWrxseAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAT9klEQVR4nO3de7BlZX3m8e9DcxMloqGVO02gYwk4cWIL4sxUiIbIJdjUiFU6Fy5RGSxJqqbUgMOYTGW00pYxBiMlYRzESyqETMakFWYQMaQGKRJARdMaQoMNNLTcAqigIS2/+WO9HTfH/Z6zT+/dF7q/n6pdZ631vu9a79p7n/2s9a61z0lVIUnSOLts6w5IkrZfhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMiZ1YkkuSvHdk/u1JHkjy/SQ/neRfJbmjzZ+2Dbu62ZKsSXL8tu7H1pLk+iRv3QbbTZJPJHk0yd+0ZXPfT99P8jMLrOeQVm/J1um5FhK/J7FjSrIOeDGwEfgR8E3gU8ClVfX0mPq7Ad8FXlVVt7Vl1wGrq+qirdXv7UmSApZX1dpt3ZdJJbke+ExVfbxT/rPA+4FfBHYD7gYuBy6qqh9Nsd1/A/wx8JKqemLc+2lrW+i50GQ8k9ixnVpVewOHAquA84H/2an7YmBPYM3IskPnzE8sya6b025n0o6+t9rvYJLDgb8G7gVeVlXPB94IrAD2nnL1hwLrquqJNj/u/aRno6rysQM+gHXAL81ZdgzwNHB0m78ceB/ws8ATQAHfB74E3Nnq/qAt2wN4PkPIbADua22XtHWdBXwZ+DDwD61sD+B3gXuAB4BLgOe0+scD64F3Ag+2dZ490tfnAB9iONJ9HLhhpO2rgBuBx4DbgOMneR6A/wZcyXBG9T2GD7AV87Qt4Igxy+fbrxcAnwceAh5t0weNtL2e4Uj+y+25PaJt51zgjtbmYtpZfmvzq8C3Wtk1wKEjZScAf9eeo48CfwW8tbM/nwGuWuB98/r2vDzW+vrSkbIDgD9r+/Zt4Nfb8rcAP2Q4Y/0+wxnFM95Pc5/P3usLLGv1dm31FnrP3dBei0dbn05qZe9v/flh68NHgTC8Px9s2/w67XfBxzzviW3dAR9b6IUdExJt+T3A29v05cD72vQzfjnHrQP4c+APgecCLwL+BvhPrewshqGtXwN2bb/wvw+sBl7IcKT6OeB3Wv3jW/3fZhj2OBl4EnhBK7+4fUgdCCwBXs3w4Xwg8EirvwvDh+QjwNKFngeGkPhha7sE+B3gpnmew15IzLdfPw28Adirlf0p8Ocjba9vr8FR7XnarW3n88A+wCEMH8IntvqnAWuBl7b6/xW4sZXtyzCkc3pbz39uz2kvJL7DSBCPKd90sHBCW99vtG3v3p7rW4HfbPM/A9wFvG7k9b9hZF3L+Mn302hI9F7fZ7Rj4ffcPwFva+t4O3A/Px5Gv370uQBe1/ZhH4bAeCmw/7b+Xd3eH9u8Az620AvbD4mbgAvb9OVMGBIMwwf/SDtibsveDPxlmz4LuGekLO0D5/CRZccB327TxzMcSY9u70GGs4RdWtnPjen/+cCn5yy7BjhzoeeBISS+OFJ2JPCDeZ7DnwiJhfZrzDpeDjw6Mn898NtjtvOvR+avBC5o0/8HeMtI2S4MYXoocAYjIdf6tp5+SPwTLXw65e8Frpyzrfvaa3Xs6Ovbyt8DfGLk9Z8oJBZ4ff+53YTvubUjZXu1tvuNPNejIfEa4O83vce29O/gjvJw3HjncyDDcNBiHcpwdLkhyaZluzCMb28yOr2U4Zf21pH6YTji2+SRqto4Mv8k8DyGI+Q9GYa8xvXjjUlOHVm2G/CXE+7Hd+Zsb88ku87px3zm3a8kezEMaZzIMPQEsHeSJfXjC8Ojz1OvX89r04cCFyX50Eh5GF7HA0bXVVWVZNy6N3kE2H+e8gMYhn82re/ptr4DGQLmgCSPjdRfAvy/edbXM9/rO2qS99w/P29V9WSr9zzGqKovJfkow1nMIUk+C7yrqr67Gfuw0zAkdiJJXsnwC3/DZjS/l+Gobt95PlBrZPphhqPFo6rqvkVu62GGYaHDGa45zO3Hp6vqbYtc56wstF/vBF4CHFtV30nycuCrDB/sm9SYdj33Au+vqj+aW5BkOXDwyHxG58f4IsNQ2Cc65fcDLxuzvvsYXvtvV9XyRfS9Z77Xd9Qk77n5/MTzXFUfAT6S5EUMZ2zvZjiDUod3N+0EkvxUkl8BrmC4JfAbi11HVW0AvgB8qK1vlySHJ/mFTv2ngf8BfLj9QpLkwCSvm2BbTwOXAb+X5IAkS5Icl2QPhouvpyZ5XVu+Z5Ljkxy02H2a0O5tG3sm2ZPhw36+/dqbIUQeS/JC4Lem3P4lwHuSHNW29fwkb2xlVwFHJfm37W6yXwf2m2ddvwW8OskHk+zX1ndEks8k2YfhQ/OUJK9tt7C+k+FD+kaGawHfTXJ+kue05/7oduCxKAu8vqP1FvWeG+MBhmsntH19ZZJj2749wY8vtmsehsSO7XNJvsdwRHYh8HvA2VOs7wyGi5bfZLib5H8x//DF+QwXPm9K8l2GI9mXTLitdwHfAG5mGB77AMM48r3ASuC/MFzgvZfhaHBLvZfXMHzob3qczfz79fsMF+0fZrj+83+n2XhVfZZh369o2/pb4KRW9jDDLayrGIaSljPcNdVb150M10+WAWuSPM5wt9ItwPeq6nbgPwB/0Pp/KsNt1E+1obJTGa6xfLuVf5zh7qPNMfb1HVNvse+5URcBp7cv+H0E+CmGgH+UYVjtEYY7ozQPv0wnSeryTEKS1GVISJK6DAlJUpchIUnq2qG+J7HvvvvWsmXLtnU3JOlZ5dZbb324qpaOK9uhQmLZsmXccsst27obkvSskuTuXpnDTZKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR17VBfptOzy7ILrlpU/XWrTtlCPZHU45mEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUteus1hJkhOBi4AlwMeratWc8rTyk4EngbOq6isTtn0X8EFgaVU9PIv+ajaWXXDVotusW3XKFuiJpC1l6jOJJEuAi4GTgCOBNyc5ck61k4Dl7XEO8LFJ2iY5GDgBuGfafkqSFm8Ww03HAGur6q6qegq4Alg5p85K4FM1uAnYJ8n+E7T9MPAbQM2gn5KkRZpFSBwI3Dsyv74tm6ROt22S1wP3VdVt8208yTlJbklyy0MPPbR5eyBJGmsWIZExy+Ye+ffqjF2eZC/gQuA3F9p4VV1aVSuqasXSpUsX7KwkaXKzCIn1wMEj8wcB909Yp7f8cOAw4LYk69ryryTZbwb9lSRNaBYhcTOwPMlhSXYH3gSsnlNnNXBGBq8CHq+qDb22VfWNqnpRVS2rqmUMYfLzVfWdGfRXkjShqW+BraqNSc4DrmG4jfWyqlqT5NxWfglwNcPtr2sZboE9e7620/ZJkjQbM/meRFVdzRAEo8suGZku4B2Tth1TZ9n0vdSOZLHf0fD7GdLm8RvXkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktQ1kz/LIWnr8M+RaGszJCRpC9hR/ge8w02SpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLv8znbST8F+fanN4JiFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUNZNbYJOcCFwELAE+XlWr5pSnlZ8MPAmcVVVfma9tkg8CpwJPAXcCZ1fVY7Por6Sty9tvn72mPpNIsgS4GDgJOBJ4c5Ij51Q7CVjeHucAH5ug7bXA0VX1L4C/B94zbV8lSYszi+GmY4C1VXVXVT0FXAGsnFNnJfCpGtwE7JNk//naVtUXqmpja38TcNAM+ipJWoRZDDcdCNw7Mr8eOHaCOgdO2BbgV4E/GbfxJOcwnJ1wyCGHLKbfknZwix3mAoe65ppFSGTMspqwzoJtk1wIbAT+aNzGq+pS4FKAFStWzN3uojhuKknPNIuQWA8cPDJ/EHD/hHV2n69tkjOBXwFeW1VTBYAkafFmERI3A8uTHAbcB7wJ+Hdz6qwGzktyBcNw0uNVtSHJQ7227a6n84FfqKonZ9BPaZtz+EPPNlOHRFVtTHIecA3DbayXVdWaJOe28kuAqxluf13LcAvs2fO1bav+KLAHcO1wBy03VdW50/ZXkjS5mXxPoqquZgiC0WWXjEwX8I5J27blR8yib5Kkzec3riVJXYaEJKnLkJAkdRkSkqQu/8e1JG2Htpcv93omIUnqMiQkSV2GhCSpy2sSO7ntZdxT0vbJkJC0XfNAZttyuEmS1GVISJK6DAlJUpfXJLRTcpxbmoxnEpKkLs8kdgAeFUvaUjyTkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktQ1k5BIcmKS25OsTXLBmPIk+Ugr/3qSn1+obZIXJrk2yR3t5wtm0VdJ0uR2nXYFSZYAFwMnAOuBm5OsrqpvjlQ7CVjeHscCHwOOXaDtBcB1VbWqhccFwPnT9leSJrXsgqsWVX/dqlO2UE+2nVmcSRwDrK2qu6rqKeAKYOWcOiuBT9XgJmCfJPsv0HYl8Mk2/UngtBn0VZK0CKmq6VaQnA6cWFVvbfP/ETi2qs4bqfN5YFVV3dDmr2M4K1jWa5vksaraZ2Qdj1bVTww5JTkHOAfgkEMOecXdd9891f5srmmOOBbbdm57bV0749Gl79EdW5Jbq2rFuLJZnElkzLK5ydOrM0nbeVXVpVW1oqpWLF26dDFNJUkLmEVIrAcOHpk/CLh/wjrztX2gDUnRfj44g75KkhZhFiFxM7A8yWFJdgfeBKyeU2c1cEa7y+lVwONVtWGBtquBM9v0mcBfzKCvkqRFmPrupqramOQ84BpgCXBZVa1Jcm4rvwS4GjgZWAs8CZw9X9u26lXAlUneAtwDvHHavkqSFmfqkACoqqsZgmB02SUj0wW8Y9K2bfkjwGtn0T9J0ubxG9eSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpayb/T0LSjm3dqlO2dRe0jXgmIUnqMiQkSV2GhCSpy5CQJHV54Xo74EVBSdsrzyQkSV2GhCSpy+GmGXHISNKOyDMJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKlrqpBI8sIk1ya5o/18QafeiUluT7I2yQULtU9yQpJbk3yj/XzNNP2UJG2eac8kLgCuq6rlwHVt/hmSLAEuBk4CjgTenOTIBdo/DJxaVS8DzgQ+PWU/JUmbYdqQWAl8sk1/EjhtTJ1jgLVVdVdVPQVc0dp121fVV6vq/rZ8DbBnkj2m7KskaZGmDYkXV9UGgPbzRWPqHAjcOzK/vi2btP0bgK9W1T+O60CSc5LckuSWhx56aDN3Q5I0zoL/dCjJF4H9xhRdOOE2MmZZTdQwOQr4APDLvTpVdSlwKcCKFSsmWq8kaTILhkRV/VKvLMkDSfavqg1J9gceHFNtPXDwyPxBwKahpG77JAcBnwXOqKo7J9gXSdKMTTvctJrhwjLt51+MqXMzsDzJYUl2B97U2nXbJ9kHuAp4T1V9eco+SpI207QhsQo4IckdwAltniQHJLkaoKo2AucB1wDfAq6sqjXztW/1jwDem+Rr7THueoUkaQtacLhpPlX1CPDaMcvvB04emb8auHoR7d8HvG+avkmSpuc3riVJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeradVt3QHq2WbfqlG3dBWmr8UxCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUlara1n2YmSQPAXdv635I0rPMoVW1dFzBDhUSkqTZcrhJktRlSEiSugwJSVKXISEtIMmPknxt5LFsM9ZxWpIjt0D3pC3K/ychLewHVfXyKddxGvB54JuTNkiya1VtnHK70lQ8k5A2Q5JXJPmrJLcmuSbJ/m3525LcnOS2JH+WZK8krwZeD3ywnYkcnuT6JCtam32TrGvTZyX50ySfA76Q5LlJLmvr/GqSldtqn7VzMiSkhT1nZKjps0l2A/4AOL2qXgFcBry/1f3fVfXKqvo54FvAW6rqRmA18O6qenlV3bnA9o4Dzqyq1wAXAl+qqlcCv8gQNM/dAvsojeVwk7SwZww3JTkaOBq4NgnAEmBDKz46yfuAfYDnAddsxvaurap/aNO/DLw+ybva/J7AIQwBJG1xhoS0eAHWVNVxY8ouB06rqtuSnAUc31nHRn58Jr/nnLIn5mzrDVV1+2b3VpqCw03S4t0OLE1yHECS3ZIc1cr2Bja0Ial/P9Lme61sk3XAK9r06fNs6xrg19JOWZL8y+m7L03OkJAWqaqeYvhg/0CS24CvAa9uxe8F/hq4Fvi7kWZXAO9uF58PB34XeHuSG4F959ncfwd2A76e5G/bvLTV+LebJEldnklIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqSu/w8n2ctDUDEaxQAAAABJRU5ErkJggg==\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"a = clf.estimators_[0].coef_\n",
"b = clf.estimators_[1].coef_\n",
"\n",
"fig, ax = plt.subplots()\n",
"ax.bar(np.arange(a.shape[1]), (a - b).ravel())\n",
"ax.set(xticks=[], xlabel=\"Feature\", title=\"Difference in Learned Coefficients\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That said, the assumption backing this entire process is that the distribution of the data is relatively uniform across partitions. The parameters learned by the each member of the ensemble should be relatively similar, and so will give relatively similar predictions when applied to the same data.\n",
"\n",
"When you `predict`, the result will have the same chunking pattern as the input array you're predicting for (which need not match the partitioning of the training data)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:43.522433Z",
"iopub.status.busy": "2021-10-14T16:48:43.521362Z",
"iopub.status.idle": "2021-10-14T16:48:43.541644Z",
"shell.execute_reply": "2021-10-14T16:48:43.542541Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
8.00 MB
800.00 kB
\n",
"
Shape
(1000000,)
(100000,)
\n",
"
Count
31 Tasks
10 Chunks
\n",
"
Type
int64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array<_vote_block, shape=(1000000,), dtype=int64, chunksize=(100000,), chunktype=numpy.ndarray>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds = clf.predict(X)\n",
"preds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This generates a set of tasks that\n",
"\n",
"1. Calls `subestimator.predict(chunk)` for each subestimator (10 in our case)\n",
"2. Concatenates those predictions together\n",
"3. Somehow averages the predictions to a single overall prediction\n",
"\n",
"We used the default `voting=\"hard\"` strategy, which means we just choose the class that had the higest number of votes. If the first two sub-estimators picked class `0` and the other eight picked class `1` for the first row, the final prediction for that row will be class `1`."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:43.547243Z",
"iopub.status.busy": "2021-10-14T16:48:43.545609Z",
"iopub.status.idle": "2021-10-14T16:48:45.846757Z",
"shell.execute_reply": "2021-10-14T16:48:45.846176Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([0, 1, 0, 1, 1, 0, 1, 1, 1, 1])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds[:10].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With `voting=\"soft\"` we have access to `predict_proba`, as long as the subestimator has a `predict_proba` method. These subestimators should be well-calibrated for the predictions to be meaningful. See [probability calibration](https://scikit-learn.org/stable/modules/calibration.html#calibration) for more."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:45.850534Z",
"iopub.status.busy": "2021-10-14T16:48:45.848794Z",
"iopub.status.idle": "2021-10-14T16:48:49.557227Z",
"shell.execute_reply": "2021-10-14T16:48:49.556350Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LogisticRegression(random_state=0)\n",
"clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n",
" subestimator,\n",
" classes=[0, 1],\n",
" voting=\"soft\"\n",
")\n",
"clf.fit(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:49.567984Z",
"iopub.status.busy": "2021-10-14T16:48:49.567418Z",
"iopub.status.idle": "2021-10-14T16:48:49.869146Z",
"shell.execute_reply": "2021-10-14T16:48:49.868612Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.54570593, 0.45429407],\n",
" [0.01064334, 0.98935666],\n",
" [0.99805699, 0.00194301],\n",
" [0.18461578, 0.81538422],\n",
" [0.05551117, 0.94448883]])"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"proba = clf.predict_proba(X)\n",
"proba[:5].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The stages here are similar to the `voting=\"hard\"` case. Only now instead of taking the majority vote we average the probabilities predicted by each sub-estimator."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regression\n",
"\n",
"Regression is quite similar. The primary difference is that there's no voting; predictions from estimators are always reduced by averaging."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:49.873921Z",
"iopub.status.busy": "2021-10-14T16:48:49.871387Z",
"iopub.status.idle": "2021-10-14T16:48:50.203642Z",
"shell.execute_reply": "2021-10-14T16:48:50.204464Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
160.00 MB
16.00 MB
\n",
"
Shape
(1000000, 20)
(100000, 20)
\n",
"
Count
10 Tasks
10 Chunks
\n",
"
Type
float64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X, y = dask_ml.datasets.make_regression(n_samples=1_000_000,\n",
" chunks=100_000,\n",
" n_features=20)\n",
"X"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:50.208613Z",
"iopub.status.busy": "2021-10-14T16:48:50.207515Z",
"iopub.status.idle": "2021-10-14T16:48:52.747643Z",
"shell.execute_reply": "2021-10-14T16:48:52.746584Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LinearRegression()\n",
"clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n",
" subestimator,\n",
")\n",
"clf.fit(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:52.756447Z",
"iopub.status.busy": "2021-10-14T16:48:52.755883Z",
"iopub.status.idle": "2021-10-14T16:48:52.997330Z",
"shell.execute_reply": "2021-10-14T16:48:52.998144Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 44.38678047, 233.09727664, 172.47190872, -76.0014212 ,\n",
" -157.78432688])"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.predict(X)[:5].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As usual with Dask-ML, scoring is done in parallel (and distributed on a cluster if you're connected to one)."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:53.005664Z",
"iopub.status.busy": "2021-10-14T16:48:53.005091Z",
"iopub.status.idle": "2021-10-14T16:48:55.834947Z",
"shell.execute_reply": "2021-10-14T16:48:55.834331Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"1.0"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.score(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The dangers of non-uniformly distributed data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, it must be re-emphasized that your data should be uniformly distributed across partitoins prior to using these ensemble methods. If it's not, then you're better off just sampling rows from each partition and fitting a single classifer to it. By \"uniform\" we don't mean \"from a uniform probabillity distribution\". Just that there shouldn't be a clear per-partition pattern to how the data is distributed.\n",
"\n",
"Let's demonstrate that with an example. We'll generate a dataset with a clear trend across partitions. This might represent some non-stationary time-series, though it can occur in other contexts as well (e.g. on data partitioned by geography, age, etc.)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:55.838348Z",
"iopub.status.busy": "2021-10-14T16:48:55.837860Z",
"iopub.status.idle": "2021-10-14T16:48:55.840917Z",
"shell.execute_reply": "2021-10-14T16:48:55.841290Z"
}
},
"outputs": [],
"source": [
"import dask.array as da\n",
"import dask.delayed\n",
"import sklearn.datasets"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:55.845508Z",
"iopub.status.busy": "2021-10-14T16:48:55.845010Z",
"iopub.status.idle": "2021-10-14T16:48:55.848164Z",
"shell.execute_reply": "2021-10-14T16:48:55.848688Z"
}
},
"outputs": [],
"source": [
"def clone_and_shift(X, y, i):\n",
" X = X.copy()\n",
" X += i + np.random.random(X.shape)\n",
" y += 25 * (i + np.random.random(y.shape))\n",
" return X, y"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:55.851187Z",
"iopub.status.busy": "2021-10-14T16:48:55.850737Z",
"iopub.status.idle": "2021-10-14T16:48:55.870247Z",
"shell.execute_reply": "2021-10-14T16:48:55.870813Z"
}
},
"outputs": [],
"source": [
"# Make a base dataset that we'll clone and shift\n",
"X, y = sklearn.datasets.make_regression(n_features=4, bias=2, random_state=0)\n",
"\n",
"# Clone and shift 10 times, gradually increasing X and y for each partition\n",
"Xs, ys = zip(*[dask.delayed(clone_and_shift, nout=2)(X, y, i) for i in range(10)])\n",
"Xs = [da.from_delayed(x, shape=X.shape, dtype=X.dtype) for x in Xs]\n",
"ys = [da.from_delayed(y_, shape=y.shape, dtype=y.dtype) for y_ in ys]\n",
"X2 = da.concatenate(Xs)\n",
"y2 = da.concatenate(ys)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's plot a sample of points, coloring by which partition the data came from."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:55.873337Z",
"iopub.status.busy": "2021-10-14T16:48:55.872577Z",
"iopub.status.idle": "2021-10-14T16:48:56.336485Z",
"shell.execute_reply": "2021-10-14T16:48:56.337061Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEWCAYAAACaBstRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAACEOklEQVR4nOydd3Qc1fmwnzszW9SbJcuSLMtVbrjKxsbgAg422IDpkEAIJaRDCARSPgjp+aVAElJJQu8JvdoGDMYY94J7t2VZvdctM3O/P2YtS9pVLyvZ85yjI+2Ue9/Z1c47961CSomNjY2NjU1HUMItgI2NjY3NwMFWGjY2NjY2HcZWGjY2NjY2HcZWGjY2NjY2HcZWGjY2NjY2HcZWGjY2NjY2HcZWGjYDDiHEl4QQK8ItR2cRQnxFCLGmD+dLFkLsE0K4A68/EkLc1lfzdwchxHlCiH1t7M8UQtQKIdQemGuDEGJCd8c5U7CVxhmCEOKoEKJICBHVZNttQoiP+liOTt04hRBZQggphNBObpNSPiulvLB3JOwfCCEeFEI8081hfgA8LqX09IRMvUngMx518rWU8hMpZXaT/UeFEAub7M+VUkZLKY0emP73wM96YJwzAltpnFlowJ3hFmKg0hNPtX2FEMIF3AR0V/H0Kk0fBsLIG8ACIcSQcAsyELCVxpnF74B7hBDxoXYKIc4RQmwUQlQFfp/TZN9HQoifCyE+FULUCCFWCCEGtTZRYEVxOHDskYBJaRzwD2B2wLRQGTh2iRBiqxCiWghxXAjxYJOhVgd+VwbOmd1ytdIduYUQ/xVCFAbOXd3UTCGEeEII8XchxDtCiDrge4HVmtbkmCuFENtaeQ+ShBBvBK5rAzCyxf4/Ba63WgixWQhxXmD7YuBHwLWBa94e2H6zEGJP4DoOCyG+1tr7D5wNVEop81psHxkwx1QJIV4XQiQGxn5bCPGdFvJ9LoRYFuK6Tq7+bhdC5AshCoQQdzfZP1MI8ZkQojKw7y9CCGeT/VII8S0hxAHggBDi5Ge8PXC91woh5gsh8gLHPw1kAm8G9t/bcgUqhEgLvNflQoiDQoivNpnvQSHES0KIpwLv3S4hRM7J/YGV2GbgtF699hhSSvvnDPgBjgILgVeAXwS23QZ8FPg7EagAbsRakVwfeJ0U2P8RcAgYA0QEXv+mlbmigGogO/B6CDAh8PdXgDUtjp8PnIX1EDMJKAKWBfZlARLQmhzfOEZ35QZuAWIAF/BHYFuTfU8AVcCcgGxuYDdwUZNjXgXubuV9eAF4KfB+TARONL124AYgKSD33UAh4A7sexB4psV4S7AUjwDmAfXAtFbm/hbwdottHwVkmBiQ6eWTcwDXAOubHDsZKAOcIcY++Zk8HxjnLKAEWBjYPx2YFbiuLGAP8N0m50tgZeCzi2iybVSL/4m8lv+/IWTQAq8/Bv4W+IymBOS5oMl76QEuBlTg18C6Ftf0Z+ChcH9PB8KPvdI483gA+I4QIrnF9iXAASnl01JKXUr5PLAXuKTJMY9LKfdLKRuwboZT2pjHBCYKISKklAVSyl2tHSil/EhKuUNKaUopP8e6Gc3r4PV0S24p5WNSyhoppRfr5jJZCBHX5NzXpZSfBmTzAE9i3ewJPKUvAp5rKZSwTFlXAg9IKeuklDsD5za97meklGUBuf+ApbiyW47V5Pi3pZSHpMXHwArgvFYOjwdqQmx/Wkq5U0pZB9wPXBOQ9XVgtBBidOC4G4EXpZS+1uQBfhq4th3A41gKGynlZinlusB1HQX+SfDn+WspZXngM+kWQoihwLnAfVJKj5RyG/DvwDWcZI2U8h1p+UCexlKKTanBes9s2sFWGmcYgZvXW1hO0qakAcdabDsGpDd5Xdjk73ogGkAI8Y+A2aBWCPGjwA3pWuDrQEHA9DG2NZmEEGcLIVYJIUqEEFWB81o1ffWg3KoQ4jdCiENCiGqsp1lazH28xdjPAJcIIaKxns4/kVIWhJArGetJu+n5zeQUQtwdMDdVCctUF0cb1y2EuEgIsS5ggqnEenJu7fgKrBVUS1rK4wAGBZTmS8ANQggFSwE83ZosrYyVFpBzjBDirYDZrxr4VQg5W76v3SENKJdSNlWS7f0PuEVzf0oMUNmDMp222ErjzOQnwFdp/qXKB4a1OC4Ty5zRJlLKr0srkiVaSvmrwLblUsovYJmm9gL/Onl4iCGew3JGDpVSxmH5PUQbxzely3IDXwQuwzLbxWGZPGgyd9D8UsoTwGfA5VhPsq3dWEsAHRjaQi5rAst/cR+W4kmQUsZjmcJCXrewHNsvY0X6DA4c/04LWZvyOZZJriUt5fEDpYHXTwJfAi4A6qWUn7Uydmtj5Qf+/jvWZz5aShmL5Z9pKWdny2u3dXw+kCiEaKokO/o/cJJxwPZOynRGYiuNMxAp5UHgReCOJpvfAcYIIb4ohNCEENcC47FWJZ1CCDFYCHGpsMJ7vUAtcDI0sgjIaOoYxXrKK5dSeoQQM7Fu5icpwTJ1jWhluu7IHROQrwyIxHoi7ghPAfdi2fJfDXVAwAzyCvCgECJSCDEeK5qp6dw61vVpQogHgNgm+4uArMBTP4ATy3xVAuhCiIto23G7AYgXQqS32H6DEGK8ECISK8z0fwFZCSgJE/gD7a8yAO4PXNsE4Gas/6mT11YN1AZWmN/owFhFtP4Zt7lfSnkcWAv8WgjhFkJMAm4Fnu3AvCcV8nQsP4tNO9hK48zlZ1hOTACklGXAUiyHbBnWTXGplLI09OltogTGyQfKsezZ3wzs+xDYBRQKIU6O/U3gZ0KIGiyfy0tN5KoHfgl8GojGmdV0om7K/RSWGeMEloN7XQev71Ws1c2rAVNca3wbyxRWiOVUf7zJvuXAu8D+gAwempts/hv4XSaE2BIwvdyB9d5UYCnWN1qbOOCLeIKA/6UJTwe2F2I5je9osf8pLGXYkVDdj4GDwAfA76WUJxMu7wnIV4O1wnwx9OnNeBB4MvAZXxNi/6+B/xfYf0+I/ddjrRTzsT6fn0gpO6oELsUKCMlv90gbhJR2EyYbm84ihDgEfE1K+X64ZWmNQLDDJ8DUjjqchRBfBm6XUp7bxjFZwBHAIaXUe0LWcCKEWA/cGvD32bRDf0issbEZUAghrsSysX8YblnaQkpZArQagNCSgMnqm1ihq2cMUsqzwy3DQMI2T9nYdAJhlV35O/AtKaUZZnF6DCHEIix/SREhQohtbE5im6dsbGxsbDqMvdKwsbGxsekwp71PY9CgQTIrKyvcYtjY2NgMKDZv3lwqpWxZOeL0VxpZWVls2rQp3GLY2NjYDCiEEC0rLQC2ecrGxsbGphPYSsPGxsbGpsPYSsPGxsbGpsPYSsPGxsbGpsOEXWkEylNvFUK8FXidKIRYKYQ4EPid0OTYHwa6cu0LJCPZ2NjYnPH4/X4KCwupqKjo9bn6Q/TUnVidvU5W+PwB8IGU8jdCiB8EXt8XqBJ6HTABq37++0KIMbJnGsvb2NjYDEh2797NunXrEEIgpSQ2NpbFixcTHR3dK/OFdaUhhMjA6rz27yabL+NUh7MngWVNtr8gpfRKKY9gVdec2Uei2tjY2PQ7CgoKWLduHbqu4/f70XWdiooK3n33XXqr2ke4zVN/xCpl3bSGz+CTndACv1MC29NpXjo6j+ZNhBoRVsP7TUKITSUlJT0utI2NjU1/YOfOneh680LDUkpqamp6zVQVNqUhhFgKFEspN3f0lBDbQqpSKeWjUsocKWVOcnJQQqONjY3NaUF9fX3I7UIIPB5Pr8wZTp/GHOBSIcTFWM1gYoUQzwBFQoghUsoCIcQQoDhwfB7N20tmcKq9pI2Njc0Zx7BhwygtLcUwmrt2TdNk0KBW2813i7CtNKSUP5RSZkgps7Ac3B9KKW/A6kZ2si3mTcDrgb/fAK4TQriEEMOB0VgtLW1sbGzOSMaPH09kZCSqqjZu0zSNadOm4fP5esWv0R+ip1ryG+AlIcStQC5wNYCUcpcQ4iWstpw6Vj8DO3LKxsbmjMXpdHLllVeya9cujh07hsvlQtd1Nm/ezJYtW3A6ncyZM4cRI9pqv945Tvt+Gjk5OdIuWGhjY3MmsHz5cvLy8pqZqzRNY8mSJQwePLhTYwkhNkspc1puD3f0lI2NjY1ND1BfXx+kMAB0XWfbtm09Nk9/NE/Z2NjYnNYYhsHevXs5ePAgiqIwbtw4Ro4ciRChgkQ7Rl1dHYqiBCkNgOrq6u6I2wxbadjY2Nj0IaZp8s4771BSUtKYY1FSUkJeXh7z58/v8rhxcXGYZnDbeiEEqampXR63JbZ5ysbGxqYPOX78eDOFAZYJ6dChQ91KyHM6nUyZMgVNO7UWEELgcDiYMmVKd0Ruhr3SsLGxOWMwTZPc3FwqKyuJj48nMzMTRenbZ+cjR44EZXGfJD8/n4SEhJD7OsK0adOIjY1l27ZteDwe0tLSmDFjBjExMV0esyW20rCxsTkjaGho4PXXX6ehoQFd19E0DbfbzbJly4iIiOgTGSoqKjh06FDIfYqi4Ha7uzW+EILRo0czevTobo3TFrbSsLGxOSNYu3YttbW1jXb/kwX+Pv30UxYuXBjynOLiYrZt20ZVVRWDBw9mypQpxMbGhjy2I6xZsyakoxqsG/6wYcO6PHZfYSsNGxubM4KjR48GOYqllBw9ehQpZVDk0rFjx/jggw8aTUmVlZUcOnSIyy+/nPj4+E7PL6WksLCw1f1Lly5t5o/or/R/CW1sbM5oiouL2bFjB3V1dQwdOpQJEybgdDo7PU5biczG3hKMj48hqzyI1Gi084ezZs2aZr4HKSV+v5/169ezaFHne8AJIVoNiXU4HL1WK6qnsZWGjY1Nv2Xfvn18+umnzUJT9+zZw5VXXonL5erUWMOGDWtcVZxECMHQ2BT01/aC31qFyGNV1Dy7hYakhpDjtLVaaI8xY8awf//+ZopDVVWys7O7PGZfY4fc2tjYhA1T99NQWYppBEcTGYbB2rVrmz3tG4ZBfX09O3bs6PRcc+bMISoqqtEE5HA4iIyMZEZuTKPCOInDB7SyMumssmrKrFmzGDx4MJqm4XA4UFWV1NRUZs4cOP3k7JWGjY1NnyOlZOfLf2Pvm49hGgaqw8GEK7/J2CVfafQtlJeXhzzXNE2OHj1KTk5QWaQ2iYyM5Nprr+XIkSONIbdZg9PR/7yBlq15VBSGe2M5GlkbVMdp8uTJnbvYJjgcDpYuXUp5eTmVlZUkJCR0K8Q2HNhKw8bGps/Z88a/2fPGfzC8lgnI9HvZ8dIjOCNjGXn+lYD1RB8qwxnocmiqqqqMGjWq8bU0THRFgBG8qpjlyMLMrCY3N7fRFxEVFcXu3buprKxk8uTJREZGdkmOxMREEhMTu3RuuLGVho2NTZ8ipWTP6/9uVBgnMbwN7Hz5b41KIzY2lsTEREpLS5v5ITRNY9KkSUHjVlZWsn//frxeL8OGDWPo0KHt1nISqoJ6dgbG+rzmJiqHgnveCL4wLpn6+nr27t3L1q1bqaqqAqx8i/3793PllVcSHR3d7jUfP36cHTt24PF4yMzM5KyzzuqWmSuc2ErDxsamT5GGjq+uJuQ+T2VJs9cXXngh7777LtXV1QghME2TKVOmkJmZ2ey4ffv2sWbNGkzTRErJgb37SUsezBcuuahZg6JQaPOHA1iKw5TgVNEuGIE6zmoV7Xa72bFjRzMzlWma+Hw+tmzZwty5c9scf9u2bWzZsqXRN3NS4Vx11VVdigILN7bSsLGx6VMUzUFUchp1JSeC9sVmjGr2Oioqiquuuory8nLq6+tJTk4OekL3+XxBSXO6NMgvLODQs6sZfcM8RBulQoQicJw/Am1eFngNiNCarVBqampChslKKcnLy2vzWn0+H5s3b252vmEYNDQ0sGvXLqZOndrm+f0RO3rKxsamw5jlDRi7ijHzqrrVSnTqjfehOpv7JVSnm6k33hvy+MTERDIyMkKadPLz80PWj9IVyeGqAoytBW3KIqW0kvtUBRHpCDJpteVbaa/8SElJSciVjmEY5Obmtnluf8VeadjY2LSLNCX+1/Zg7i0FRYCUiIQInDdORkR13sQy9OwvoLn/wucv/onaolziMkYx6brvkjKucxFRYDm3Q4bHStBMgbGlEG16etDu6upqPvnkE/Lz8xFCMHz4cObMmRPkZHe73WRkZJCXl9dMeXQkkioiIqJVhRMVFdWBq+t/2ErDxsamXfQNeZj7SkE/dQOUpXX4Xt2D64auhaAOmTyHIZPndFu2tLS0kA5vVQpG1cdBRLBC8fl8vPbaa3g8HsBabRw5coTy8nKuuuqqoPHOP/98Vq5cSWFhIYqiNPpW2uu9nZiYSFxcHOXl5UHO/IkTJ3blcsOOrTRsbGzaxdyUH5QAhwnyWCXS40e4HeERDGulsWjxYt57423L1AQgYHx9AqlEo04JbkB04MCBoPLkpmlSW1tLQUEBaWlpzfY5nU6WLFlCTU0N9fX1JCQkdNiJvXjxYpYvX05FRQWKoiCl5JxzzunRxkh9ia00bGxs2kX6QldmBWEpk+5V9O42Q4YM4UuLLufwy+vwYzCkIYJo1YVIi0WdnhZ0fHl5ecieFqZpUllZGaQ0ThITE9Pp3hRRUVFcccUVVFVV4fV6SUxMHBCFCVtj4EpuY2PTZ6jZgyyHstnC1BPrhOj+ETbqzEoi+5sXYuwsRtZ6UYbFowxPCGm6GjRoEJqmBSkOIUSvZWjHxcX1yrh9TdiUhhDCDawGXAE5/iel/IkQIhF4EcgCjgLXSCkrAuf8ELgVMIA7pJTLwyC6jc0ZhzYvE2P/MfBI8DtB1UFRcF6aHfKmLE2Twh2fUZ1/mLj0kQyeOCtk2KuUEur84FQRzrbzKUJx5MgRtm7dSn19PampqeTk5BA/I9jpLet9SK+BiHcjhGDUqFFs2rQJwzAafQ2KohAfHz9gzUZ9hehO2Fy3Jrb+06KklLVCCAewBrgTuAIol1L+RgjxAyBBSnmfEGI88DwwE0gD3gfGSClbWzcDkJOTIzdt2tSr12JjEw7Kj+wmf8vHqE43mbMXEzVoSC/O9gTS+zT6trOQR0cikkpQc7agxF8PfKnZkd6aCt7/yY3UlxVgGjqKqhGfMpq5l/0CVXWhjEpESY7COFSO/819UOcDQMkehOOSbISrY8+yO3bsYOPGjY2rBSEEmqZxxRVXND7Vy3ofvlf2II9VghDg1nBcmo06Kona2lrWrl3L8ePHGzvenX322QMy4a43EEJsllIGhbOFTWk0E0KISCyl8Q3gKWC+lLJACDEE+EhKmR1YZSCl/HXgnOXAg1LKz9oa21YaNqcbUko2P/FLDn/4MobuQ1E0hCKYefvPyTrvkl6a9QKgKsT2RGBFsy1r/3wPx9etwDT8AKTFTCEn42ZQVFRFA0WgjE/G3FPS3LmuCkRmHK4bp7QrjWEYPPXUU/j9/mbbT64iFixYAID3X5uQRXXNzWoOBedt01GSg0NeS0tLKSsrIzY2ltTU1HbLkJzOtKY0wurTEEKowGZgFPBXKeV6IcRgKWUBQEBxpAQOTwfWNTk9L7At1Li3A7cDQeUGbGwGOiV7NnF41SsYPitc1DStJ/UNj95P2tS5OKN72nYuCa0wACqbHyklx9efUhia4mZ6xs2oSuDp3ZRgSswdRcFDGRJ5vBqzogEloe2kuerq6tCSNumOZxbWIkvrg/0wuomxPg9l6akeFrqus3z5coqKTskVExPDJZdc0u2+3acbYc0Il1IaUsopQAYwUwjRVuByKJUfcpkkpXxUSpkjpcxJTk7uAUltbPoPx9a+g+H1BG0Xikb+9jW9MKMAhreyb2TQFtkkmW1w9ARCWpAlob+9qkBWBV9bS9pKmjtZQFBWe61ExBBzmxXNiyVu2bKFwsJCdF1v/KmqqmL16tXtynKm0S/KiEgpK4GPgMVAUcAsReB3ceCwPGBok9MygPy+k9LGpp8gROhHKFrd3APcQ3BcrQu4u/n8QjBk8rlNnN5tSBRql26GNBu1xO12k5WVFVSiQ9O0xnpOypDoZsmIpw5SUIY3j5Dau3dvUH0p0zTJzc0NWXfqTCZsSkMIkSyEiA/8HQEsBPYCbwA3BQ67CXg98PcbwHVCCJcQYjgwGtjQp0Lb2PQDss69JKhuE4A0dYZMbbviatc5G/hb4HcKMAv4JxBc9iPntgdwxSSiuSIort2NIkJERWkCHGpzxeFQUKendbgsybx58xg+fDiKoqBpGk6nkzlz5pCRkQGAiHGhThsCjia3OcVyhmstcjdaW7WcrEtlc4pw+jSGAE8G/BoK8JKU8i0hxGfAS0KIW4Fc4GoAKeUuIcRLwG5AB77VXuSUjc3pSHL2VMYs+hL733saaZgIVQUks779W5yRnUs8a4uqvEPsfesxqo4fJHHUJMYt/QpRyX9t97yoQWlc8uflHFv7DpW5+6mO8JFwyGWZo0wJqoI6NRX17Az0VUcxj1Qg3BrqrIyQiXitoWka559/Pueeey4ej4fo6OigwoXa4tGI1GiM9SeQXh11dBLa3GGIiOYZ7JmZmRw6dChIQZzM57A5Rb+InupN7Ogpm9OVqrxD5G9djeZ0kXH2hUTED+r+oDUFUPQ5JaX1rPrbzzH9PqRpIFQN1eniwp8/T9zQ0Z0eVlZ5MHaXIH0G9RluykQ9kZGRDB48uF9EKNXV1fHqq6/i8/nQdR1VVVEUhcsuu2zAdtjrLv065LY3sZWGjU0HME14+5uw/UlQXbyzJ4kqb0szkSB10jks+PG/2xyqKu8QB1a+QENZIWnT5jHs3KVoTjdSSj755BMOHDjQWIMpKiqKpUuX9ouKrz6fjwMHDlBcXEx8fDxjx45tt/R5S44ePcr69eupqakhJiaGGTNmtFvUsL/SL0NubWxs+gkb/gKfPw26B8PnpcobKitaUrJ3c5vDHF+/ks/+ci+m7keaBgWff8ret5/kwl++wKGjxzl48CCGYTQ6l6urq3n//fe57LLLeuGiOofT6WTChAlMmDChS+cfPnyYVatWNV5bVVUVH330EaZpNutLPtDpF9FTNjY2YWb9n8BfD4AQElWEtkA42vCZGLqP9f/4MYbPgzStG6fhbaCu+DgHlj/Prl27gmo9SSkpLS2lvr6+hy4kfGzYsCEo0krXdTZsOL3ideyVho1Nv0ECJUAkYOUa1BTXsvnFHZzYXkhkvJvJV05gxOxeSFj1nEqWUwQMj6vicFUcpjz1XKk63YxZ/KVQZwNQeXQfUgZHIRk+L7mfvYt/ytUhzxNCBGV2D0RqakL3Pa+trbU6A/YD301PYCsNG5t+wTrgZ1iZ1yYwi9qy+3j5rtX46v1IU1JbUseqhz+lKr+GqVd2zYRiUQk8DHyIpagWwLgFsPVlCNz0pw4upkHXKKiLRo2IxfB7GTZnCeMuu63VUTV3JLKVnAZHZDRZWVns3LkzKLzV4XAQGxvbjevpH0RFRVFbWxu0PTIy8rRRGGArDRubfsBhrOS5ppnQ6zB8d+BvuAzZpAxGZEIZBbveZOKSZBzulJYDdQAduAUrL1bH8EuqT7yHa3YikfsSwFMLhhdNVZg7ooK6i/5IrSuT2PThRMS3XV0hNn0EUcnpVOcfaVQ+AKorgjGLvkTy5CkcOXKEhoYGdF1HCIGqqsyfP/+0uKnm5OSwZs2aZiY4TdPIyel8C9v+jK00bGzCzvNAS/OMn6ik48SmlVJ5PAlHhJcLf/gmqWMLMXQFzfk6VkHoe+hcHvinQCmgc2iVny1P+qyyGkYtg0bO5NyZE3AVb4BBY2HWXUQlj6WjcU1CCObe+zc+/PnN+GqrQICp+xn9hevImPkFhBBcddVV7N+/n7y8PGJjYxk/fvxp02dizJgxmKbJxo0baWhoICIigpycHMaOHRtu0XoUW2nY2ISdXKwWMc2RpkrMoBoqjycx/46VpI4rQHMaaK6TR7wOjACu7MRchwAPxbsNNj/uw/Ce2lNy4BifkMTCn37c1QshJjWTSx9ZScneTXiqyhiUPY3IxMGN+x0OR7cilLqKlJKSkhIqKyuJj48nOTm5V1Y3Y8eOZezYsZimGZRoeLpgKw0bm7AzDdgB+Jpt1ZwGlfmD0dw+hs04gupoqVg8wLN0TmkMA9zsfau+mcIAkIZJ+eEd1BbnEZ2SBLyK5feIBa7BKh3SPkJRSBk/sxMy9S5+v5933nmHsrKyxm2JiYlcfPHFvdY743RVGGCH3NrY9AOuAaKApjWa3AhlGTnXX0B0ErSegxs6Yqd15gKx1JeFHlBRHXgqTwBfwao1tQ2rweY9wGOdnKt/sG7dOkpKSppVsC0tLeWzz9psxWPTCrbSsLFpj6Mfw3/Ohf9Lgn/NgkMre3iCBOAZYAmQhNXp+C7g+4xZMIKrH7kJRQlVykKho0//p3AAT5A6OQslhJ3BNHTiMg8BJ2jumPcA/6Zl/4yBwIEDB4IitkzT5ODBg2GSaGBjKw0bm7Y4tBKevRiOfwoN5XBiPbywDPa+3u6pnWMw8ACwHPgflsnJ+noqqoqiPYBVmvzkV9aBlcvxzS7MNYixS57CEZWEop4q3Ke6Ijjrmu/gcK9DSi/G4VHoG8/BPDYisNJxAJ937fLCSGulzU3TtCvYdgHbp2Fj0xbL727MlG7EX29tH9uXpS9mA09grUiOYflBrgeaFiksA17BcnZPBC7F8kcE445L4qLfvsru1/9NwbY1RMQPYuwlN5M+bT6y/pf4nrwbWRUPpgKKiUgqwfnlpxCugRfplJGRwfHjx4O2p6WlnRahvn2NXbDQxqYtfuYAUw+97wEDuuvw9Htg3xtQcwLSZ8LQc6wch9W/skp7eKtgyHS46M+Q0ZZz+QBwG1borg9rVRIBPA2EqiPVOr7/fYa5tw7MJs+Uqh91ym4cS+6gN1s99QbV1dW8+uqr6LqOYRhomoaqqixbtuy0CfftDeyChTY2XSE6FarzgrdHDuq+wijdD4+dB3o96F5QHZAxG+JHwM5nT61wTqyHJ8+H2zdC8rhWBvsFUNfktQdLeTwM/F+HRZJSYu71NVcYAIYDY8cUHEsGlsIAiI2N5dprr2Xfvn2UlZWRlJREdna23fu7i9hKw+a0wFPt5dCao3hqvKRNTCV1fA/F4c/9f7D8e81NVI5ImHNf98f+7zVQX3zqtemHox8BH0HL/mK6Bz75FVzxdIiBfMCeENtNYG3nZJK0HqplDjyFcRK3283kyZPDLcZpga00bAY8+buKePdnq5CmxPAbaM7dpE9O5cIfzEVRu7kamH47eGtg9S/A8ILigDn3wjl3t39uW1TnQ/HO4O0hmlGaEo5XR5L7/iYchT9i5AVXk5w9tckRSuAnVMvSzuUhCEUgsuKRRystBdK4A5TRSZ0a63RGSsn+/fvZvn07Ho+H9PR0ZsyY0eEaWlJKysrKkFKSlJQ0oPI6bJ+GzYDGNEye/srLeKqbZ6ppLo3zvjGTMQusBjir8z7m6d1PUdpQwuDIwXx5/Fc4J31OxycydGgog4hEy4zUXaqOw8PtV6s1JXyUO5Syhgh0qYAQqE4XE6/4JuOXfbXJkT8CVtG8HIkLuA74TqdEM8sb8D22BfwG+E2rx7ZLw3XbdESsq/0BzgDWr18fVOrd6XRy1VVXER0d3ea5JSUlrFixAq/X21h/a+HChaSldbzVbV/Qmk9j4Kg3G5sQlBwsw/AHP53rXp2971tx+KuOf8gjW/9EUX0hhjTIr8vn4S1/YO2JTzs+kapB9OBOKwxpShqqPRh6i1VAdX7bJ2pWx7gTNdGUNbgthQEgJYbXw47//YWGytImJ/wQGIXl/I7EUhhTgds7JS+AkhiB6ztno31hJGpOGtqiUbi+fbatMAJ4vV527twZ1BtE13U+/7ztkGS/38/bb79NXV0duq7j9/vxeDy89957A6aniG2eshngtG5nF4F9T+9+Em+Lmhlew8uTu5/o3GqjE5iGyeq/rWf/qsNIU6KoCmddOpaZN0yxTGbrHmr95JRJkH0JbHiE4zVR6FINOkRRHRTtWk/WnCWBLTHAU8Au4DgwGkuJdA3h0tBy0rt8/kDl+PHjbN++nfr6etLT05kyZUpQK9qKigpUVQ3K/zBNk4KCgjbHP3r0aMjcECklBw8eZNKkSd2/iF7GVho2A5rkUYmoThV/Q/OnPs2lMfYLozClSWlDachzi+oLe0UmKSWvfv89Sg+VN24zdZPPX9uNEIKzvzwVyg+3PsDMb0HO7XDBL3D+5+ew8oVmpcYBEOCIaFl/VmDlZ0zsqUs57SkrK6OqqoqkpCTy8vJYv3594wqiurqagwcPctVVVzVTHNHR0a0mDLYXwuvxeEKeaxgGDQ0N3biSvsNWGjYDGkVVWPTDebz94IcAGD4D1aEwdNoQRp43DEUoJLoSKfeWB52bHNF2f4iukr+jiNLDwfNJE3a8uZecL05GHb4Aij4Hs3mRQlQnTDjV4W7kBVdx+KNXMHyeZocpikrqpHN6Rf72MA0d09DRnP07ZLWiooKioiIiHC6GlGtQWI+SEoU6eTB+RfLOO+9QXl6OEALDMJBSNlsFmKaJ3+9n27ZtzJlzakUaHR1NWloaJ06caFaeRNO0diO0hgwZgqIoQWVNNE3rdz6N1gib0hBCDMVaT6dihX08KqX8kxAiEXgRqwDPUeAaKWVF4JwfArdi1ZG+Q0q5PAyi2/QzUselcMN/ruDw2mN4qr2knTWYlDGDGkNuvzjuBv6145/NTFQu1cUN477c5rhSSva+f4gdb+zBV+cnMyed6dedRVRiZJvnnfi8sHnkURNM3cRX7yNi9vdg62PgqTwVMeWIhLO/CxEJjccnZI1j6pfvY+uTv0HRHEgkiupg/g//iar1ToXW1vDX17LxPz/l+LrlSNMgPjObmV/7OYkjulfmXEqJ1+vF6XT2SBSRlJIPP/yQo0ePWgZKXeKQgkVlQ4kVbvTVR1kzoY7S0tKgm3dLTNPkxIkTQdsXLlzI6tWrOXLkCEIIXC4X5513HsnJbT+IDBo0iMzMTHJzcxtXNJqmkZKSQkZGRlcvuU8JW/SUEGIIMERKuUUIEQNsBpZhldcsl1L+RgjxAyBBSnmfEGI8VreamUAa8D4wRsoQMYpNsKOnbABWHF3Oc3ueodxbTpJ7EF8YdiF7yndxsPIgSe4krhv7Rc5NP6/ZOZ8+upG97x9E91r/YkIVuGNcXPPIJbjbcAp//sYePntsc0jFobk1bnn+WoQioDIXPnoQDq2wkgXPuQcmfQlC5Jf46qop3r0B1RXB4PEzUbQeiODqJCvv/yLlh3di6qcitDR3JEseepvIpM5lnZ/k4MGDfPbZZ3i9XhRFYfz48cycObNbymPPnj189tlnzR3VEuJ0J5eVZWEgeT71IGZrmr0F6enpLFmyJOQ+v9+Pz+frVEtXKSUHDhxgz549SCkZPXo0Y8eORVWDfVfhpN9lhEspC4CCwN81Qog9QDpwGTA/cNiTwEfAfYHtL0gpvcARIcRBLAVi1ze2CaKs1suGQ2U4VIVzRg/iwqxFXJi1CFOaHK46xA8/ua9x5VHnr+NPWx6mylvJkhGXAFBf0cCeFQcw/KeeRKUh8dX72fXefqZfc1arc4+am8WGp7aFjOo6+8tTLIUBEJ8JyzpWbtwZFUvGjIUdvfwep+LoXiqO7W2mMABMXefAiueZfP1dnR4zLy+P1atXN97cTdNk165dGIbRzBzUWfbs2RMU2YSAWs1PjerDbWqWGaoD9/j2TE4OhwOHo3MKXAjBmDFjGDNmTKfO6y/0i5BbIUQWVnzgemBwQKGcVCwnGyGnY4WFnCQvsC3UeLcLITYJITaVlJT0mtw2PYMhDfaW72FHyef4DF/7J7TDf9cf4/KHV/Pbt3bzq9d3suR3H7H+oOUMV4TCM7ufDhlN9czupzFM60ZfergcxRH85Gf4DPI/b9uBHhkfweL/Nx9HhNaoIIQimHnDZCYuGZitP2sKjyGU4PfD1H1U5u7v0pibN28OurkbhsHevXuDb/qdoDUntZBgIHFIhXg99EoxIiICVVUblcGsWbMazUZSSnJzc1m+fDlvv/02+/bta9e8dToSdke4ECIaeBn4rpSyuo0lXqgdIdeXUspHgUfBMk/1hJw2vcPBygP8/LOf4jE8CAQSyXenfY/ZaV1z8h4squEvK/bja5EX8YMXtvHW9+cT5dI4VBm6j4Jf+qn0VpAUMYjoQVFII/iGIBRBbGpMu3JkTBnCTc9cQ8mBUoQiSB6V1P3s9DASnzkGabTsYw6q08WgMVO6NGZNTegGUkIIGhoaiIlp/30OxciRI9m6dWuQ8nBIhTjDCYrg7MTRvFcTnJE/b948Bg0aRENDA3FxcWjaqVvk+vXr2b17d6NCKyoqYv/+/SxZsmRAZXR3l7BeqRDCgaUwnpVSvhLYXBTwd5z0e5wszpMHDG1yegbQToaUTX/Ga3i5/9MfU+GtoEFvoF6vp0Fv4A+bf0dhXdtP86Y0WXPiE36x7qf8av0v2FCwHikl727LRw91sxfw6X5r1ZkSOThof+Aoop3WjSpxWDwJmfFBN3rVYeVbdARVU0gdl8Lg7EgUdT1WL4qB+WQamzac1EnnojaJmBJCQXVGMOqCa7o0ZlJS6LIkQggiI9sONmiLs846q9kNX0GgScF5tWkIp4ZIiqQoSw15o1+3bh2RkZEkJSU1UxjV1dVBGeC6rlNSUkJubm6XZR2IhDN6SgD/AfZIKZtmOr0B3AT8JvD79SbbnxNCPITlCB8NbOg7iW16mk2FGzFb5h9g2bY/zH2fL467IeR5Ukp+s+HXbCvegsewQlG3FW9l/tAFePXzMUKsLaWkcfVx/dgv8n8bfx0UTbUoazEu9ZTZ4uIHzufDh9dw4vMihCJwRTuZ9+1ZJGbGd+IqXwYewvqqSawkvEeAEZ0Yo38w566H2PXyPzj4wUsY3gZSJ5/L1Bu+jys2of2TQzBjxgwKCgqa3Yg1TWP69Ondcgo7HA4uv/xyjhw5wokTJ4iOjmZMTBoRNRKRFIEyLJ6DL7wQ0rRUU1NDXV1dUEJfQUFBSEe3ruscO3aMrKysLss70AineWoOcCOwQwixLbDtR1jK4iUhxK1ALnA1gJRylxDiJWA3oAPfai9yyqZ/U+uvxfB6m7fGBnSpU+WtavW8XWU7mykMAI/h4cPjH3DLyPNwb1XxtHBCG6bk7JHWk+2k2CncKG/hw/wPODE4F5ySi4cv4cvjv9LsHHesi4t/cgGeGi/+ej/RyVGnnNgdYjdWaXJv4AegAfgW8BZBF97PUTUnk669g0nX3tHqMVJKZH4NZnEdSlIEYmhcq1FFbreb7Oxsjhw5gsfjISoqihkzZjBqVNcy2QsLC1m/fj3l5eVER0eTk5PDvHnzQh7bVqRTqH0ulyvkdkVRiIiI6JK8A5VwRk+tofX4hQtaOeeXwC97TagBSlW9j39+eJAPdxWiKoKLJ6dz6/yRuJ39+6aUnWciTR1amICcXpNp8a1HJ20paq4wTiKlxO88wrxxI1i9p5gGv4EiwKEpfHXBKJJj3Rz+LJdVD3+KUBQmMYuJ5kzm3TWL0ROHtzqfO8aFO6YrdZf+xyll0SglVt+LbcD0LozZN3i9XoqKinC5XKSkpHQonFT6DHzPfo4sPOmrEIhEN84vT0FENI8wOnToEKtWrWr2tF9bW8vOnTsZOnQoLlfn3u+CggLeeeedRj9GRUUFq1atwuv1MnZssDkxOzubLVu2BPk94uPjQ5rGhg4dGtKcJYQgOzu7U7IOdMLuCLfpHj7d5NZ/raewqgE9YJd5cf0xtuaW869bz+7X7SzjXv+IOf5y1s5OxOeyvpDS0KjzprDr/RrOvu3UsRWect47+h5Hq47gNbxoQkOXzSNsVKES7YjiwSvOYuPhcj7cVYjLoXDxlHSyh8RSV1bPhw99iuEzsPJDLT5+aB3p/0olMr6nnxgrCR2rIYDQTuD+wPbt29m0aVPjTdLtdnPxxRe3WyJD//AwMr+apvZBWVKP/539OK88lQDo8/n46KOPgsxDUkpKSkpYtWoVixcv7pTMGzZsCFIAuq6zYcMGsrOzg74HkyZNIi8vj5KSEkzTRFVVNE1j4cLQYc2qqrJkyRLefffdRnOalJJ58+adcd3/bKUxwPl4TxFltd5GhQGWIjlUVMu2YxVMzUoMo3RtIw2Da/6bz9r4pXizChCqgad8LJ7ybJ5SFCYcLOXsUYM4Vn2U+1Z/H7/px2/6cSiOIIVxktlpcxBCMHNkEjNHNne0Hl6b22qm9uFPc5m4pKefGBdgud1aror8wJQenqtnyM/PZ/PmzRiG0XgT9vv9vPPOO1x33XVtPoQY2wsJciiZEnNPKVLKxnNPnDiBoighQ2OllOTl5eHz+XA6O57xXl4eXLYFrFpPtbW1QZFYqqqydOlSioqKKC4uJioqiqysrDZ9KYMGDeKGG26guLgYXdcZPHhwM2e5aZrs3bu3MU9k5MiRTJo0qVPXMRA4c+LETlN2naiiwReiNLhhsr+g/z7NAkQuu4z81CxKfJOpOnIZlQevwFM+HlDxmIIX1x0D4K9b/0K9Xo/ftEI+/aYfgUBBIVKLJEKLJMoRxQOzHyTS0XrUjd+jY4aIrDINE93T9byA1lkEDMfq1w3WCsMNfA2I76E5qrCKI6zB6uDXPZqGlDaloaGBdnOeQkUggBWF0KTyRHurXyEEfn9weG9btNXDYvPmza3Ok5qayqRJkxg5cmSHnO9CCAYPHkx6enozhQGwatUq1q1b11gEcfv27bz++uut5o0MVOyVxgBnaGIkbkew49ehKgxJ6N8OOtd55+G/YDFqiAgqgIo6H4ZpsK9ib9A+icShOLhv5o9QhMKEpAloStv/zkOnDWHLSzsC5qlTKKrC0Om9USzOCfwbeAfrxh4LXEXP+TL+hxWZddJfIIA/AV1va+rxBPuKwLpZ+nw+pEfH3F+K9BkooxJRmpj0lFGJmPtKg1ZzYmgcook/ID09PWR58JO43e5Oh9xOmzaND97/IKSX9PChQ8yfP79T43WWiooKjh492kxBGIZBTU0Nhw8fZvTo0b06f19irzQGOIsmDcGhimbfFVURxEQ4OGf0oLDJ1RGEEEz/xQ8wQ0SfCCDCqbKjeF+rT6ZO1cnUlKlMTp7crsLANEn2f8pl0/7DBSP+QXrMTkBaJdQXjiQpq2tho+3jAi4H/gr8mp5TGPuBP2KtLuoCP7XAnQQ73ztOVlZW0BM0WKaXpDoH3ofX4n/nAPqKQ/j+thH/R0caj3EsGgWRDqvTH4CmgFvDsbR5uQyHw8HChQtDPtlrmsa8efM67YsbprfeZtU0ez+/t6ioqNWQ3Pz80yudzF5pDHCi3Q4eve1sfv7qjkZz1JRhCTxw+VloAyAD2e1U+e7F4/nje3vxNK3zBBzUX+eBtTtBCV6JOBUnC4dd2LFJpIRXvgT73iTZX8egOMiK20y+6zLUJX8k7azWkv36M28Q2hwlscqxze/SqGPHjmXv3r1UV1c3q8J6ds5MxCv7rPavTTA+O446MhFlaBwizo3r22djbC/Ed6KSXGcN9V4PKc+tY5DXhUiNRls0CjUlmszMTL70pS+xf/9+8vLy8Hq9DBo0iIkTJ5KQ0AUFXuUlzR9FvqOu+WpDwtCI3vfrtVawUFGUdtu/DjRspXEaMDw5msdun02dR0cIiHQNrI91Wc5QkmJc3Pf8Vk4+FGqRhbgSd4ISbF93Kk4mDJrADeNu7NgEuWtg35vgrwOse4qGh0zjDUj7GYiuVWjtC6Rp+QNE0ANALaGzyyVWLkjX0DSNZcuWsX//fo4ePYrb7WbChAkMqlDwKxXBJ/hNjO2FKEOtCCLh0qgc4eat3ZswfTqmNFGcgiFEMvdIGuY/NlKYcozEpecSlzGKSZMm9Ui3OiUthll1qbwTexRdmOiKRDMFDlRmj5/W7fHbIyMjA4fDEeSLURTltAvJHVh3F5s2iXIP3I/TNCURTpW6QBlyV/yBkArDIRxcOfpqrh/3xY4Pvv/tRoXRHAkH34NB/a/aqPTq+N89gLmrGEyJSIvBsTQbZfDJp9YFwAcEKwgdmNGtuTVNY/z48YwfP75xm1Fa3GrkWdMaXVJKVq5cic8XWAUJMIUk31nPoYgqxjTEk1KYzvIf38DEL97BmEWd+BzbQKTHEpOayOUnVI46qqjQvCSYbka4k4mcnNkjc7SFoihceumlrFy5ksrKSoQQOJ1Ozj//fHulYdPzSMPAyM1FxMSgDurffojeIinaRTPTswxt01YVlZSolJD7WsUVB4ozuEueUMHVui08nPie/RxZUNMYkSRP1OB7fCuub81ExLiA84BpwBYsxSGw/CdfBXr+f6g+qgZNN4L9zA4FdcKpz6OysjJk21JDkRwIKA2AJMdwtj79WzJnLcIdF7oGVWfw+Xwcmu6kIcYkJS+aUfogtIkpaHOzEFrfmGljY2O58sorqa2tRdd14uJaz4YfyNhKI8w0rHyfynu+j6yrQxoGrpkzSPjbX1FbKeZ2ujIhI47kGBd55fWYEryV2UQmfw5q89WGaZpMdE3C3+DHEdHBPgaTvgirfx7amjPu8u4L38OYBTXIotrgEFbDRN+Uj2PBcKwYloeA1cBKIBKrh1nP9gf3VJez+rffpPLYXjLjZ3NW0uUoigOBAIeCkj0IZeQpn0FbUVEnnwMkElPqKKpKwbZPGD5vWbdkLCgo4N133wWsiCU1WmXo0KFccMHIZlFbfcXptrJoia00woh/zx4qvvFNZJMnM++69ZTd8GVS3n07jJL1PUII/nLTDO59fitHSmpRjVT0srNxD96AIgSKECQcT+bsDfN58+n3kRKGzUhn3ndm44pqJ3kqfhgsewJevxmaRlld+wq4+182ryxvCNm9D0Mii2ubbFCxzFQLek2WNQ/dSfnhXUhD50jxRxRX7iIzcQ7Dci4k7rxJiGHNn6YTEhJwOp1BuR6qKRjZYK3qpDQortuL4nKhOLqX+GaaJitXrgyqPnv8+HEOHz7c5TpWNq1jK40wUvvvx5C+FiYTXUc/cAD/nj04xo0Lj2CtUFLt4bm1R9lytIKMxAhumDOccek9d9NNiXPzxNdnk19RT51XZ3jy+ZR6itlUuAGzSFD8VC16kxyLY5tOsPxXH3PpL7/Q/uATr4HRF8PRVZbiyFoADnf75/U1FUcRu/4Ovguw8jyaoCmI9L4zp9WVFlB+cAfSOHVDrvOVsKfwNcoO53PBjcHKSgjBwoULeffddzENE8Mw0KQg0e9idF0sftPLuty/Y0odIR2kTQ1dULCjlJSUhEye03WdvXv32kqjF7CVRhjRj+dCqGxRTcMoLAyb0jBKS6l/5VXMgkKcs2fhvuB8Cqp93PSPtTT4DXRDsr+wmk/3l/CzqyZz3ugkpM+HEsi3qPX42XK0ApdDYXpWYqdDf9MSTiV2pUalsnTkpax6ey2FenWz40y/SfH+Uqryq4lL68DN1BUN2Zd0Spa+oK60AFP3E615EP+ageKvR8gEJFNABBSbABwK2rSeSUL01lTgqSpH9zZgeBtIHDkRzdU8X8ZXW4nQNPAH5314qspaHTs1NZXrr7+egwcPUldXR6oai7p5J9sKn6Gobg+m8KM63Zx758NoWvcUd1vmMJvewVYaYcQ1Zw6+zVugRRau9PlwTOxZ23Sz8Rsa8Lz/AWZVFa4556ANP1Xh1bveMo9J0wSPB/Hss2hjx/LojT+hzqs3OqulBI/f5DfPrWP4s/eg+H1oWVms+daD/Hm/H00VSGllpj90wzQmZMR3S+aqEzVW+GkLVE2hpriuY0qjn1FTcJQ1D3+X6vyjCCFwawbnpOgMitBx8gC6+AqGXALCjTI6DW3RKERkx/tRyxov+ro8zGOViMQItNlDMRM11v3th5zY9CGmoYOUKJoToapM+8qPGHX+VY3nx6aPDDmuojlImza3zbndbjcTm/4P54wjofRcCrZ/goab1OJMeLcK7zufINJjrciwlKjWB2yFlJSUkNVnNU077UJd+wv9P/vrNCb6yzeixMZCk8b0IiKCqC/fiJqc3Ctz+rZto2BaDhX3fJ+qB39K0cILqbz/AasPgmlS/vVvIuvrGxWZrKvDv2sXG3YXECqxttYQVChuMAwOlzfwp89r8OomdV6Dep9BVYOf7z69Oaj9amcZMiEFJUQUjOE3SMyK79bY4cDQfbz/4I1U5u7H9HsxfB7q6v2syk3Ho6sI/Djkv3DLZbiVZTiXDELpRFkYs7IB7983YmzIs/pb7CrG98RW9v75z5zYvApT9zfWgzJ1H4a3gS2P/4zSA3/DqmcFqsPJtK/8KNCtz/JbKA4nzug4xl1ya6evOWrQEEaefzVD9qfCoSowJUiQedX4Ht+CrO987SxFUfjCF76ApmmNGeaappGRkcHIkaGVnk33sFcaYURJSCBlxXvU/PkRPCtXIuLiiPnqV4m48opemU8aBmU33Yysbm7mqX/hRVxz56JlpCNra4NP9HiIrqmgPCo4lFMKQYTfcuS/P+Y8dCW4NIQhJesOljJ3bCdDZZtw1qVj2bPiAL46HydLVWkuleyFo3qhpDlY+Q6C3mqUVLD1E3RPQ7NCfgBSCo5UxjFuUHnTjZ0ODdZXHQWvfiq3QgJ+k/T6bHb4Q9+cDb+fA8sfZdDoZ4GfAgsZOf8KYlOz2PvW49SXFzJk8nlkX3Rjl7v1yWNVyCovQU8ghkTfVojjnM7nVKSlpfHFL36Rw4cP4/F4SEtLY/DgwWEJd/X5fGzcuJHDhw8jhGD06NFMmzYNhyP0CtHr9bJnzx5OnDhBbGwsEyZMIDGx/1amBltphB01OZn4n/8Mfv6zXp/Lt2ULMkRBOllfT/1zzxP7ox8E3cROcnnxVv45ZlGzUh+a4Wd67nYi/daY9c5IzBBKQ0qo83avimxkfARXPnQxG5/dzvGtBbiinUy6dCzjFvV0IbijWH2+tmMtxM8H7gN6NsqqobwI0wx+TwypUKc3+Vpqbhh/NTg7bropP7Ib5+4TaDL4RuVUI3FrcXj0EJ0RJTRU+rFqV/0Eq05WAsljp5E8tmeyqs2KYEUJgG4iS0IlYHYMt9vdLBkxHJimyWuvvUZ1dXVjr5CdO3eSn5/PsmXLgpRYQ0MDL7/8Ml6vF8MwyM/P58CBAyxcuJDMzN5PSOwqttI4k/C2vvyXHg/aqFEoKSkYx4412yciIrhk0XQqUobxwrpjOFQFv24wLv8A3179eONxM3K38tmI6Xgao5IkIDAMSc7w7j89xaREc/5dc7o9TutUATdjleiQWIkdHwJHgOdovdFk50kaMwURYjxNU0mJ0a2Vhe6FURfB0n90eNytT/+WAyueZ/7Qe4l1Dwk+QAh0M3QlW9UJ6dNPKn0BrAJ6dtWrDG5F+TkUlC5GhkkpKSoqorq6moSEBJJ7yLRbWlrKwYMHkVIyfPhwUlPbLjdz7Ngx6urqmjWXMgyDyspK8vPzSU9Pb3b8li1b8Hg8jcdLKdF1nY8//pgbbrih3yYG2krjDMKZMz3kU56IjCTyissRQpD0n39RevW1SJ8P6fcjVBXX+QuIuu4avqmq3HDucI6U1JIS6ybi959QrynIQLmdnNztjCk9wpHJXlwpG1EcDRi+WGbFX0VybD8Mbw3iTawigE3fIx3Iw2rPOrWD43iAEiCZU700mpM4fDypk86h8PO1GD7rJq46XEQPySLjJ6uh6hDEZkBMiBt/K5Qd2smBFc9j+DwcKFvJ5CHXoilN2qaqAm+8iXSIoEK4igMiBwlGLji5OpFYzaJ6FiUtFpEWgzxRDXrgfRaAS0PtQuFIr9fL22+/TWVlZeO25ORkLrroopDVejvK1q1b2bp1a2P+x549e8jOzmbOnNYfWoqLi0P2ATEMg9LS0iClcezYsaDuhWA1vaqpqSE2tn8Gd7T7rgohhkspj7S3zab/I9xu4v/4MJXfuQNpGOD3Q0QEjmnTiFh2GQCOceNI3biehhUrMUtLcc6cgfOsU/26YyMcTM607NnyZz9Fzcyk9tFHkZVVRM6cycIbnbxQ/jG6tFY1qrOa7d5n2VSYSU5q92oi9T6HaL2s+DHaVxom8DfgeSzTlglcB3yLUDEn537vTxxY8TyHPngJ0+9n2LlLGXvJzSjuKIju7MrM4Pj6xzEC4bG5lZ8R40plZOJ8TGmgOtyomQkkXDOB8+Y8wq7XHqWm4AhClbhiKsk6T2HUQgea++TTrcQqVdLzOL84Cf3joxjbCkE3UcYk4fjCSEQXCm2uWbOG8vLyZjff4uJiNmzYwDnnnNMl+aqrq4P6h+u6zr59+xg9ejQpKaF9c3FxcWiaFpzYqKpBnQOBVjv6mabZqg+kP9CRT+llrCI3TfkfPdcYwKYPibz4IvD/nop77gXTBMNAP56LfugQjrFjAcscFXnZpe2OJRSFmNu/SsztXwXAkAavvn19o8I4ic/w8syep/q10ig9XI7hTyZ5tAtFaak4JNCRJLFngBdornhexPKHfDnoaEXVyL7oRrIv6mC13lY5BnwdRSlCCNm4mNxV9CoHSlcQHz2ckcuuJ+tiKxlvyJRzGTLl3Cbn/x54LSC3wGrqdCvQG42pQDhUHAtH4ljYvegm0zQ5cuRI0NO6YRjs37+/y0ojNzc35HZd1zl69GirSmPEiBGsX7++mdIQQuBwOBg2bFjQ8WeddRaffvpp0PGpqalEhOgx019oVWkIIcYCE4A4IURTw2Ysra25bfo9RnExlffcC/X1gQ0G5rFcSq+6htRNGxDurn+0Df4GvEboJ/WCuoIuj9ub6F6d937xEUX7SnBGRnHlH1XcMQJFPWmicgInvwrt8RTB/cA9wBPAOUAWPW8RlsBdQCmZ58Det8FoorN9Rh1l9QeYc+78Nsa4B6s17UqsaLHFwMDIcWgtua87LVZVVW21N0ZbLWGdTieXXXYZq1atoqzMSn5MSUlhwYIFIc8bM2YMpaWl7N27F0VRkFISFxfHBRdc0GXZ+4K2/oOzgaVYzYybptHWYJXS7DZCiMcCcxRLKScGtiViPZ5lYYWyXCOlrAjs+yHWI5AB3CGlXN4TcpxJ1P/vZcs01QLp9+NZ+T4Rlyzt8tiRjkjcmptaf3DYbnp0RpfHbYopTRTRc+lFG5//nMK9JRg+A92r8srd13POrasZNuMoqsMNLMEyL3XEKRkiIgmAauAWrK/bj4GevCkcBooBSXymwsSrHOz4r9+SViiAg5lf+wXu2PbMXWcFfjpOfVkh21/4IwXbPsEREcXoxTeQvfiGPisSqCgKKSkpFBUVBe0bOnRol8fNyspi7dq1QduFEO2WJYmPj+fyyy/H6/U2lkdvDSEEc+bMYerUqZSWlhIZGcmgAVDlulWlIaV8HXhdCDFbSvlZL83/BPAXrEe0k/wA+EBK+RshxA8Cr+8TQozHMhBPwFo3vy+EGCOlPL26tvcyRlEReINXA9Lvxygt7dbYilC4fuyXeGr3E81WHE7VyY3jb+ryuFJKXjv0Kv/b/19qfNWkRg3hlgm3MSttVrfkBdj3/sFmPcPrSmNY+X9LUDSFW1+6DqVTJVBGYPlFQhFY2fETIIOee5L30NRfMv4yJ5mzNfK3GCjqENJnPENEfM/fiLw1Fbz3gyvx1VYhTQNvdTmfv/AwVcf2cfY3ftnu+bm5uWzbto36+nrS0tKYNm1al6rDzp07l9dffx3DMKw6V5qGpmldNk0BREREsGDBAlatWtW44pBSMnv2bOLiOhZ67XK52j8oQGRkZL8OsW1JR9bKZUKID4DBUsqJQohJwKVSyl90d3Ip5WohRFaLzZdxqlflk8BHWIHylwEvSCm9wBEhxEFgJlZvS5sO4jpnNvXPP4+sq2+2XSgKzhnd9zksHXEJLtXFC/uep8JTTlp0GrdMvI2pKR2NPArmv/tf5L/7X2pURIV1Bfx+8//xY+2Bbo0LYPhDZ6pLU1plSzqV2/d92u/R7cPye/ykMwO3QTYthYxOURizOAL4Er3RWwPgwIoX0BvqkOYphWt4PRz99G0mXv1toga1HvW1c+dONmzY0GjLr6mp4fDhw1x11VWdVhwJCQlcd9117N27l/LycpKTk8nOzm7zCb8jjBgxgvT0dI4dO4aUkqFDhxIZGdn+iWcAHVEa/8L6NvwTQEr5uRDiOaDbSqMVBkspCwJzFQghTnqd0oF1TY7LC2wLQghxO3A7MKA0eF/gXrgQLXss/j27ocGyv4uICFwLL8A5sSN2+7YRQnBh1iIuzFrU7bEAdFPn5QP/C/KV+Awfz+55uttKI3N6Gkc+y23MMgdAQPLoJFRHZ7PBc8D4K9T/Elx5oOmgtLS5m0DH/TtSynbi9TUsBfT/sMKDdSACazVzdaek7wzFezY2Rmo1RdUcVB7b26rS0HW9mcIA6xr9fj9bt27lvPM6H7HldruZMmVKp89rD5fLxZgx/a+rY7jpiNKIlFJuaPGP27303q4R6psT0gsmpXwUeBQgJyfHLoPZBKGqJP/3RWqffIr6l19BOB1E3fAlIq/uvRtMd6jx1aCboS2Q+bUnuj3+7JunU7CzGL/Hj+41UJ0qqqYw71tdMH3VlcK/r4K6YtAa4HtnQUj7/vAQ204hpeTg+y+y879/xVNVSlRyGpO/dA/DZl/UyhnzsRTH41jmqi9guf6697TdFrFpwynevbFZ2XQA09SJSg75LAdAVVVVSCUopSQ/P7/H5Wwc35Qg6POEufz8fDZt2kR1dTVJSUnMmDFjQPgt2qIjSqNUCDGSwA1aCHEVnXlU6jxFQoghgVXGECwvH1gri6berQyg9/7LTmOE203M124n5mu3h1uUdol1xuJQNPwtW7UCGTFdd3aeJDo5imv/fin7PzhE8YEyEobFM27hKCIohqPbYdBYiG6ZdFYI/AfYACRhhdPOh+Xfg6pcMP2WFWpDMcweHKKh0tvAN7ACEYM5sPw5tj33BwyvVdOrriSf9X/7IerxT8lIj4dh50H6zCbj/hl4iVNmsWcDf3+3y+9Le2RfdCOHP3oVo4nSUDQH8ZnZxGe2/nQeERERMqENeqfjnVnRgP/t/cgjFSAESnYSjovHINpr3NUDHDlyhA8//LAxkqu+vp6CggKWLl3aatjuQKAjXr5vYZmmxgohTmD9J36jF2V6AzjpNb0JeL3J9uuEEC4hxHBgNNa31qYFR0pq+dXrO/nafzbw9/f3U1bblo29j5Cy1bpWbaEqKtdmX49Lbe5YdKoubhwXnPvQFVxRTs66dBwX3H0u0y4bScS718JfsuH5y+DhYfDm16BxtVOM5St4AzgBfI5lGnoK9rxiKYyTFHsgZHVfE1gRUhYpJTv+95dGhXESw+dl+1svwAc/gicvgBeWBWQ6jBVs6MF6rpOBv/8LHOzaG9IBYoZkMe++fxCVkoGiOVE0B0OmzGX+D/7Z/HpMiTROvQeRkZGkp6cHlTPXNI3Jkyf3qIzSp+N7bIulMCRgSsx9Zfie2BqyzH6Pzi0la9euDQr91XWddevWtXLWwKDdlYaU8jCwUAgRBShSypqemlwI8TzW2nqQECIPa439G+AlIcStQC4Bw6yUcpcQ4iVgN5Z57Ft25FQwGw+X8f3ntuA3JIYp2X2iilc35fHE12aT1onS2j2Gpxre+y7seN66oWbNgyV/h0EdtxUvG3U5bs3NS/teoNJbSUZMBrdMvI2zkif1vLzv3QWHVoDusX4APn8GEkfBOfdA1S8hpqqF79kD/j8SlKMRpbVinvIAoZsYGX4v/rrqkPvqfKr1Hpp+OPw+bH0MpjuxItBbomP1D++9znWDJ8zkkj+vwFtdjupy43CfqislG/z439mPuacUpLR6ZlySjZIcxfnnn8+HH37IiRMnGpXHrFmzyMjombDskxg7i8FnNDdimxJZ48M8XI46KqlH52uKruvU19eH3FfazSjFcNORMiLfa/EarID0zVLKbd2ZXEp5fSu7QgaySyl/iVWC1CYEUkp+9fquZpVo/YaJ4TH5xwf7+dlVPfsk1wGB4OkLoXAbnHRkH1kF/54FdxyAyI59aYUQXDT8Yi4afnHvyQpg6LD9yVPK4iT+elj3J6jJh2kfgBoiAdIwIdkF+U1uFLm11vagxK4IYEpIEVSHC2d0PN7q8qB90c4mJjp/PWz+N0y/C0uDtXQzKkDHwz67ihACd1zzz1FKie/p7VbV2sATvcyrxvfYFlzfORtnpJPFixfT0NCAx+MhNja2zaS5riJL6iFUdJxhIssaelOfoqoqqqoGlRQB+nW2d0foiHkqB/g6VqRSOlZU0nzgX0KIe3tPNJvOUlnvp7QmuIKpKWH9odbbc/YaJzZA8c5TCgMAad2Utz7W9/K0h+EDo5UifQ3lsOmfUOkJbWZTFag9ea4AoUKRAicMkE1v3m6sVKOZIacRQjDp2jtRW7ReVYXJ5OSS5gdLHVjYysUobezrXWReNbKsHowQPTO2nnKHRkREkJCQ0CsKA0BJjQZniFucqiCSO98lsFNzKwoTJ04MKpqoaRpTp3Yv4i/cdERpJAHTpJR3SynvxlIiycBc4Cu9KJtNJ4loI0Q0ugvF4FrD8Bsc35rPsY15+BvaqIRaujf0dr3BWn30N5yRkBTKbCYgfhhIEz4tCn561U04VgPVTZTGvPth2eOQuR7Ed7FKkWQD3wYeoa0M81ELryHnlvuJHDQEoajEuAzOSc8nPaZJvwlHJEy5GSsP4wGsVUVk4MeFlXne+aqxPYEsbwh9ebqJLOp6z4zOooxPBpfWXBZVIBLcKMPje33+nJwcxo4di6qqjUmHU6dOHfBtaDtyJ8nEigU5iR8YJqVsEEL0Aw/r6YU0TepffInaxx5D1tTiXnQhMXfegdqBbl5up8p5Y1P4ZG8x/iZPeW6HwnWzggumdYWCXcW898tVjQ/b0jCZ+61ZjJ4XIow0uZW8D0ckDOmn9S6X/gOevcjqZSENq2a4IwLO+iKs+T84Vgtv58LioaAIUAUcqoZXmhR9jkiAeQ80iW66ms7mTIyYfzkj5l9uvchdC88sAjPCUrjOaOv9m34y+m0RMBv4FMuAP4eebhrVGURKVOhg+G70zOiSHA4V123T8a84iLmvDBSBMiHFqqjbB6G3iqJwzjnnMGPGDBoaGoiKiuq1VVVfIlor+NV4gBD3A5dzKorpEqzQkT8Aj0opv9SrEnaTnJwcuWnTpnCL0WEqf/BDqz5UQyB6xuFASU5m8Ifvo4Qor9ySOo/O95/fwq68Khyqgk83uXhKGvcuHY+idO+L4m/w8/TNL+NvaFH62aly9Z+XEjekhXxSwuNz4cTGUyYqoYA7wfJpRHStZWivU7wb1v4eindBxkzLAe6MgYeHWr4EsNboiS6oN6C+yfvhiITzfwmzv9uzMtWXWcEEtQUwbB6MWNiKk71/4H1yKzKvutFEJQEiVNzfmY1w2218BgJCiM1Sypyg7W0pDWGp4wwgBTgXa6G3Rko5YO7CA0lp6CfyKTpvblBtKBERQcx99xLz1ds6PFZuaR0FVQ2MTIlhUEzPOEQPfHyET/6+PkhpKKpgylUTmfHFEI52by2svBc+fxp0H4xcCBc9AokjqCmpw1PpISEzDq0d81n+ziI2PL2NyrwqYlNjmHHDZIZO7Z3S3a1ycDm8dBWgWN8EU4cpt8DeV6CmwHLsz70fzv5OiNyM/oGh+6grPoE7NhFndO+tRqTfQF91BH3LCUyvl8LaXewsfg1XWjJz7nyImFS7UkN/p0tKo8mJ/dSW0D4DSWk0vPceFXfehawNrhLruuB8Bj31ZBikOsXu9/bz2WOb0b3BIZ4Tl2Yz56ut166SUlLd4CfCqWE2+Fnxm9UU7ytF0QTZcSuYmfk2Dr0MBo2DRX+AURc2npu3rYDlv/wIvUlhQUVTSBwWjzQl6ZNTmXz5eCLjezkqRUrIW2+FuyaMgOxLwWUlpB3+9CCfv3EAT7WPjKlpTLt6IpHhCHFugwMrnmfbc38AKTENnYwZF3D213+J5uodOb21lbz57S/gb2jy/ywU3HGJXPrXD1C13k+ws+k6rSmNjqwT1wkhZkgpN/aCXDZNUFNTrcZILdE0tGFZfS5PS9InDwkZOKS5NYbNaD3G/tP9Jfz2zd2U11nloq85UYurpAFTN5mY8CbTU17D4Q+4zUp2WolrN7xr5XSApah8zRWVqZuUHrLCUiuOV7F/1WGu+uMSohJ7qahcZa7l66g8BopmmdsW/AzmfJ/NL+5g28s7G5VpTdEBDq85xtWPLCUirn+0njmxeRVbn/lds6TBExs/ZIO4n3Pu+H2vzHn0k7cwW5QZQZro3gbyt3zM0Jlf6JV5m2KWN2B8XggeHWVMEsrwhH7be3ug0BGj6ALgMyHEISHE50KIHUKIz3tbsDMRx+TJqEOHBsX1C4eDqJt6Jvu5O8QNiWHikuxmpiTNrZE+KZX0yakhz9mVV8mPX9pGUbUHvyFx1PtRCuswdRMFnWmpb+BQW5QI0Rvggx83vqzMa61PhYWpm3hrfWx7eVfXL649nlsKpfvAXwfeKits+KMH8e16h63/29ls9WUaJt56HzvebCV6LAzsfu3R4Cxzv5fjG1biayWZsLvUFec19j9viun3U19W2CtzNkXfUYTvHxsx1uRibDiB/8Wd+P+7q9XGTTYdoyNK4yJgJHA+lhN8Kc2bMtn0EEIIBr3wHM4ZOeB0IiIiUFJSSPz3ozhGda81Zk8x6yvTWPz/5jNqbhZZs4ey4M5zuPAHc1t9envyk8N4m4SounUTI3CsS6tFEa0k9TcJ143ogJlHGpJjG/I6cSWdoHg3VByyoqma4q/HXPMwqhb8NTL9JnlNchLCTX1ZcKMiAEVR8dZU9MqcyWOnobmDV35CVUka2bmGT51FenX0N/dZ4dAnS4b4TcxD5Zh7B3ZGdrjpSBmRYwCBEuX9Y619GqOmpJD88v8wSkqQdXWomZl91gmtKYcqD/LR8VXops6c9POYkDShUTGkT0olfVLolUVLcsvqm0VfVro1RGCLV4/GlCpWFHcLmuRLTLvmLD77z6aQvpSmtDRh9RieCsskFQJNr8AIVV9KQHRK/+m/kDxuOrlr30G2MH8KVSVqUO8EFKRPX0BUcjo1hbmYgTLqqtPNoNFTSBrdu9UJzKOVVkh0S/wmxs4i1HHJvTr/6UxHyohcihVem4ZVrW0YsIeONU226SJqcjIk980/ttR1av72d+qeeBJZV8eKm6fw1rgG/NKyR7+fu5L5Qxfwzcnf7rQ9eGJGLCdq8jAMDdMfg6EINqZFMzO/Fs3U2Fa0hKmpb+JQmpioHJFw/s8bX467cBT+Bj9bXtyB7jMwQxYBBEeko/MX3xFSpzYpWNgEzY026SqSjydSvL8Ms0lhPs2pMnnZ+N6RpwucdfW3ObH5I3RPPSebh6hON1O+9H0UrXfeN0Vz8IWfP8/u1x7l2KdvI1SVkQuuInvpTb3vV1BF6/mT2sDPlQgnHXGE/xyYBbwvpZwqhFgAtFYzymYAUnHHd/GsWIFsaKAs0cEbIyvR5anVjdfw8tHxVVyQ+QXGJo7t8LjbS7az1/lb4rNrAYnuSaTqyBKODYlnVk46GceqOVZ+EymJmQzzPo1oKIXE0bDoIRhxqvyYROKbVUttRiHuikj4e0RwjT4BQ6e03i2uWzgjYfHDVuFFfwMgQYuAmDSY8U0WTXbx/u8+oXBPMYqqIFTBuV+byeDs/vM0G5M6jMW/+R87//dXSvZuJjJpCOOX3U7a1LmtnFEKHMHqRtCxVWUoHBFRTL7+LiZff1eXx+gKSlYrOUAOBXVK16/HpmMht5uklDlCiO3AVCmlKYTYIKUMXTynnzGQQm7DgX78OEXzFjTmhnx8XiIvX56G39XcJCYQXD76Cr4y4ZYOjVtcX8y3Pvh6s457UgqEEcNXhv2Gy3Myg582pQzKbzClya/W/4LPS7bjMTwoKExaM5OMo8ORTaxajgiNq/+0lJjBPd+ToZHjn8H6P1s5GWOWQs7XwHUqobGuvB5frY/YtNiQfo6BgY7VlHMFVhMnH1a2+a/oiwKIPYl5rBLf8zsAaVWjB9SZ6TgW9g//YH+nOyG3lUKIaKw6y88KIYoJaYS26Ut2l+3mzUOvU+YpI2fwDC4evoRoZ+dvmP49exFOJzKgNBy6RIR4kFCEglPp+E1j+dH3MFqYdISQRLj9jBhWjhDNy5r4TT/Lj77HqtwPUBWVRcMWMz/zfDYWbGhUGAAmJtvOWU99bB1nHZyOv0EndWwy59yW07sKA2DobOunFaISI3sv5LfPeAJ4H0tZnDQZrgMeBn7QrZFN3Y/ubcARGdM3ZTyGxeP63mzMfWVIr44yMhGln+XODEQ6ojS2A/XAXVjdZ+KAXv522rTF8qPv8e8dj+IzfEgkhysPsfzou/xpwSNEO9svNdIUbVgm0n/qGWDy9mpeuCa4XacqVOYNnd/hcYvri9BlcFloKSUVnubROqY0+cmn93Ogcn/jyuRI1RE2FW1CVdRGhdGIIjk+7RBX3HoJ56TP6bBMLfF7dQp3FaNoCqnjUwbw6qAnOdnQqSle4E3gXjoWcNkcU/ez9dnfc+j9lzANHXdcEtNv/nGf5GkIp4Z6VngKN56udChPQ0ppSil1KeWTUso/A62n/tr0Kl7Dy392/Auv4UUGopB8po9KbyVvHHq9nbODcWRn45w8GZxWdm5UvcGtj+Xi8Jm4FRdu1Y3LdHM9N1G30UtdWejGMi2ZnDwFd4i+E6Y0yU5sXuVza/EWDlYeaGbK8hpeNhVtxKt7Ea14NJ1q1zOKD316jKe+/D9W/u4Tlv/qI57+yv8o3FvS/omnPa19vj5CN3tqn03/+RmH3n8Jw+dBGjoN5UV89si9FO+284UHIq0qDSHEN4QQO7DavH7e5OcIVo9LmzBwtOoIigj+2Pymnw2FXet+m/Tk40QsWWIpDk1jum8I/8r6f3xr6h3cFHcrF/3vaqqe87DmHxt4/muvsen57e2OeV7GXJIiBuFQTkXmuFQX56afR1p085XMjtIdwasJLAWTHJkcUjkoCCYnT+n8xQLVhTWs+uNadI+Ov96Pv0HHW+Pj3Z9+iN8bvDo6s2gtFHYU0PkoK199DUc+eSMoyc/wedj5yj86L55N2GnLPPUc8C7wa5obM2uklMFtxc5w/LrJyp0FrNpdRGyEg8tzhjJxaHyPzxPjjAlp9gGId3VtPiUmhsS//Bn50O+Rfj9KlNWg5lzd5JkfvYy/rvl821/dzZCJg1nf4OO5z45S1eBn2rBEvvWFMWQOss51qS7+MO9hXj34CqvzPgIEczPmcV12cOBdgisBp+LEZzbPDNcUjeyEscS7Enhx3/OoQg3YwgX3z/4JDrVroaL7PjyMbNkgCMt0dmxDHqPOy+rSuG2jA3VADF0x8fQddwM3Y60sdKyugA7gh10azVNRgqJqmH5f0L7awmNdF9MmbLSqNKSUVVhtXe3w2nbw6ybfeHwDh4pqafAbCAEf7CrkGxeM4drZPdPH4iRp0elkxmRyuPIwJqfyAlyqi0tHXtahMXTDpKreT2yEA4emIKVke24l+RX1jBkSy6hAU7OCXUUYIdpl6j6D15/cwkvxjsbWsqv3FbPpSDl3Ls5m74kq4iOdLJmaTrQjirKGMjRF4/WDr/LpiTU8eM7PSIlMaRxv3tD5PLvn6aB5FKEwa8gsXJqbCzIXsr1kG27NzfTBObjUrkfyeGt9zXIqTiJNia8u+ObWPUzgUeBZrJtwFPAdoGOfVd8zEsuv8SywExgN3ICVntV5IpPTQpftEAqJIyd2WUqb8NFuyO1Apy9Cbt/ZdoLfvrW7WW9uAKem8NY984mN6NnkqbKGMn627kHya0+gChXd1Lko8xqOHRzPrrwqMhIjuXneSKYMax6rLqXkubXHeOzjQ/gNE1URXJEzlM8OllJQadUlMqVkWlYiv71uKvlb8/ngD5+G7M53PN7F+1nBDXVUxWqLrakCZ3Qe8SPfRJenbsQKCukxGfzl/L81i6DZWbqD3278DR7dC0hinDH86Oz7GRnf8+GRuVvyWfl/q9E9LfqCOFSufiREX5Bu8Q/gGZo7l93AT4ELQp5xurHzlX8E1b5SXREs+uWLxA0dHUbJbNqiOyG3Nu2wandRkMIAcKiCbccqmDs2JcRZXScpIok/LXiE3OpjVHorUXypfOeJHXj9hZgSTlQ0sD23gp9ccRYLxp9KZHp9cx7/WnWgmazPf3YUOFWeB2DLkXKe+OQwN52dGfKJXHGq5A0KHbp48nDdkEQmbsVv+pqlXpiYFNcXc6z6GFlxWY3bJw46iycWP82RqiOoQmVY7LBeC8scOmUIQyakULCruFFxaC6NcYtGta4wSvdDbSGkTgF3R7vP6VhP7C39NR6s1UcXlUZNARxfC1GDYeg5/agZUxlwCKt4xKmqxxMu/xoRCcnsfu1RPFXlJI06iylfusdWGAMUW2n0ALERDoQgqGy4lBDVg725W5IZO4xMhnHX05vx+IxmNZ48fpM/vLOX+eMGN958H199OEi5mSEWml7d5I3NeXx1wSjO/doM1vxzI6ZuIk2J5tZIGJnIETeNXdlaQ9HqQ/YiUoVCrb8GKSWHqg5SUFvAsNgsMmMze2Vl0RKhCBb/eD6HPz3GgY+PojoVxn5hNEOnhsgoryuB5y6B4h1W61fDZ/X/Pq8jNv56LMURii5UeZUSVt4HGx4B1WmVA4lMhps+gIQQ7Xb7DBP4DfAWVkKgH5gK/BaIRAjByAVXMnLBlWGU0aanGHBKQwixGPgTlofu31LK34RZJC7PGcoHuwqDbsgRTjXIRNQV/AcOUPPHP+HfsQNt1Chi7rzDCpMNsDOvMmRL5qp6H9UNfuIireijspqOt3T36gae9z8g8Xe/57zCevKy5sGkHEYunUpmTjqfvbiN9YfK8LVSBwrAWzUSR2QJQmkeqmlIgyFRaXx/9fc4Vn0MRSgY0mDSoMn8cOaPu+zg7gyKqjBq7nBGzW3nZvvilVCw2erSd5LVv4SUiZDdXrHn6MBPqCqyozonMMCeV2Dj36yy7Hpg9eKvh+cvhW/u6Px4PcaLwDs0TwjcghVD8/PWTrIZoPSXdW2HEEKowF+xyrWPB64XQoS9KtzEofF844IxODWFKJdKpFMlKdrJn76cg9rNvty+HTsouXgpDW+8iX7oMJ4VKym98mo8H3/ceExCVOh8BSEEEc5TzwXDUzqWk6kqgtkRXsq//g38O3cSXXqYsZseZ+xzdzNEP46iKvz8qsksnJiKU1VwqIL4SAcOtfm/k6f0LNBjcSqWfAKBS3Vxy4RbeXznvzlceRiv4aVBb8Bn+Pi8ZDsv7Huus29R71F1HPI3NlcYYPXUWPuHDgygYDm9W+aruALbO8n6v1hzN0WaUHHY6vURNp4j2ATnw8os7/iDis3AYKCtNGYCB6WUhwGEEC9ghaHsDqtUwLWzh3HRlDS2HasgyqUxZVhCtxUGQNVPf4asb5JwJSWyoYHKH99P6prVAHz5vOH8/u09zVY6Lk1hydR0nE2ynO9clM3dz21p1t/CoQmUQLFyn27idihEux1c89JvkA3Nm/bg8VD1i1+Q8u47uJ0qD1x+FvctHY/HbxDj1vjje/t4bXMeigBFCFQlkp/P/j1HvGvZULCeBFcCS0deypiEMfxr56NBocM+08eKo8u5cfxN3X7feoSGcsskFXRDBOo7mgh4KVaY7T+wTFKjsBTGlPZP9VTBhr/A3tchKgWqW+kXIlTw1nRQnt4guD2xhcRSGgOrZpVN2ww0pZEOHG/yOg84u+VBQojbgdsBMjP7roF9bISjx53evq3bQm43jh1DNjQgIiJYMiWd4iovT605jKIIdENywYRU7lpsVaSt9dXy3N5nWHNiDUOnglk1kbLjU8lIjOP280cxKiWG1zfncay0jolD47goO5GqPx0KOa9//4Fmr10OFZfDKjX9vYvHcc2sYWw6XEZspIM5o5NxOVSmcAWXj7ri1DUZPkwZ2qzVNCs87AwaG1RAEbD8CaMv7sRACwI/ncBTDf+YCrUFp0xRimb9tFz5KCoMntS58XuUGcBHQMvPNBVLYdqUlpayefNmSktLiYuLY/r06QwZ0ktVmXuZgaY0Qj26B5nzpZSPYoWnkJOT06mYYqnrNLz2OvWvvIqIcBN1/fW4Ljg/bH2FlYR4zIJgp6lwucBlPcEJIbhl/ki+eE4W+ZUNDIpxNYb5+k0/3199N0V1hY1P9s7o9UydXcavz/tt43XdMv+UA1pKSXV0NLI6uA2oOrjtOj4ZiZFktFO0z6k6GR43gkOVB5tfE4Jpg6e3eW6forngoj/D2988VRJddUFEIsy5F7Deq/JjlRh+g0EjElHUHrL4bvqHFa2lN1nlmDogrH4j/nprhaG54JJ/gdb1kird5w5gI9aKzM+phMAf03pTizOH4uJi3nrrLXTd+v7V1dVRVFTEBRdcQFZWVniF6wIDTWnkYRX4P0kGkN9Tg0vTpOzGL+PbtLnRJORd/QmRX7ye+J8+2LUxpaThzbeoe/JJZF09EZdeQtRXbkKJ7Fg11Jivf53q3/xfo6nIRPDZmFl8NPcqxFObuWhyGosnDUFTFdxOlREt/Bbr8j+jrKG0mSnIZ/o4XHWYPeW7GZ8U3EtLCEHMN75OzZ8faWaiEhERxHyvZ/oifGfKHfxwzX3opo7f9ONUnLg1N7dMvK1Hxu8xptxkdRFc+xBU5cKoRTDrTohMojy3kuW//Ij6Sg9CCBRVcP5dc8jMCS742Gn2v2n1Sm+JMwam3gKVhyE2A2Z8E1LC3Q8tA3gJy7fxOZCFVdt0RBhl6j+sW7euUWGcxDAM1q5dy7BhvRda3lsMNKWxERgthBgOnACuA77YU4N7P1zVTGEAyPp66p5+huibv4LWhaeCqvsfoP7FlxrH9B84QP0rr5Dy9lvWaqEdom69BaOwkNrHH0c4HPw15zrWjcjBgwZHytmVV8X7Owt5+IZpIf/59lfsC1nXyZAGBysPhlQaANHf/hZS16n9xz+Ruo5wu4n9/j1EXdUzYZMj4kfytwv+yfKj73K0+ihjE8byhaxFxHSySm+fMHQ2XPvfZpsMv8Gb/28lnmpvs7Xuyv9bzdV/uYTY7pZpjx6C9ZTeMo7bgKk3Q2o4zVGhSAbuDLcQ/ZLS0tA9yevq6tB1HYej96MFe5IBpTSklLoQ4tvAcqw18GNSyl09NX7DBx80dzoHEIqCd82nnVYael4edc8939jgCACPB+NYLg1vvEnk1Ve1O4YQgrj/92Ni7ryDfdsP8tmqUrxNwlw9foPth4r5+Gd/5uyzs3EvXIjQTn2sqVFDcKmuIF+BpmjNSnkEzasoxH7vLmK+823M6mqU+HiE2rNtMpMikvjiuBt6dMy+4vjWAgyfEXRPN03JvvcPMuNLU7o3waw74cDblhnqJEK18jEGn9W9sW36lIiICGpqggMVVFVF7eHvVF8woEJuAaSU70gpx0gpR0opf9mTY6uJiRBK66sqSlxcp8fzbdzY7AZ+Ellfj2fVqk6NpcTEsF1NwAxR9qXBhHVbjlBxx3cpWXoJZhPFNy9jPpqiNSsvrgiFSC2KnMHtV7gXDgdqUlKHFYaUkjqvjh4ik7y3qSmqpWhfCb763u8R5qn2IENkRpq6SV1ZCLNSZ8mcY7W9dUSCK9b6nTIBvvRuaOe8Tb9lypQpaC3uA6qqMmHCBJR+k83fcQbUSqO3ibzmassc429x01EU3As7X/JBSRoU+guuaShdiJyIi3SiqQp+o3mynMPQifNUI+vq8O8/QO0/HyX2ru8CEO2M5jfn/ZaHNz9EbrVVVXRMYjZ3T78HTenZj3/t/hJ+9/Yeiqs9aIrgsukZfOfCbBy93NzIW+tlxa9XU7S/FEVTMHWT6dedxdQre68g3pDxKUEVAAA0t8bQaWltnqt7dXa9u5+DHx9FdaqMv2g0o+cOR7QM0c75Gky6EQq3Ws735HE9eAU2fcXYsWNpaGhg27ZtgPVglZ2dzYwZA7MtkV2wsAX1b71N5ffutur5SIlwuUh68nGcU6d2em5pGBTOnIVZVNSsxohwu0lZuQJtRMdKP5i1tci6OhriErnsoY+p8zZXGi6/l7++9EMSGqxoJ3XEcFI/WR00To2vBkUoRDmiOn0t7bEzr5JvPbGxWQ6IS1NYODGV+y/vXXPK2w9+QP6OIswmZjvNpXL+985l+KyhbZzZPVb/dR0HVh9tUr9KJSEznst+s6jVLoCGbvLave9RcbzKMm9h1b0acW4mC+44p9dktQk/uq5TV1dHZGTkgPBj2AULO0jk0iVEXHA+3o2bEC4nzpycLtvyhaqS/N+XKPvKzRj5+aCqoKok/PGhDikMs6aGirvvwbPyfRACNSmJ3/74V/y//U68Ph1ZV4ciTb734T8bFYY1b+iPtTedzE98fBhfizIqXt1k5c5C7liU3VjKpKepr2ygYGdzhQGgew1WPruKz+vWkRSRxOWjrmBKSucVf1uc982zSZuUyu739qN7DUbPzWLc4jFtto09uu44lSeqGxWGJavOoU+OMfWKCcRndN4MajMw0DSNuC6YufsbttIIgYiIwD33vB4ZSxsxnJSPV6EfPIisr8cxfjyig08ZZbfchm/TJvBZ9XyM/HyG3PMNXn3tNQ7Gp1H67TsZvnk1WpNkLxERQeT11/WI7J3hWGldyPpXDlVQXO3tNaXhrfaiqErIvh+1FXXk1hwjt+YYu8t2cfOEW7l4xJIem1sIwajzsjrVtClve0FQSXZrLCjYXWwrDZt+z8DzwgxAhBA4Ro/GOXlyhxWGfuQIvi1bGhXGSaTPR/2//83EjHjm/P4BnAlxiKgocDoRkZE4Z51N9C0398ZlUO+v561Db/Lr9b/k8Z3/obCuoHHf+Iw4QlVN0Q1JWkLoMuo9QWxaLEINntgUBiXpp1J4vIaXJ3Y9FvaM86jECJQQKxGhKETE9977ZGPTU9grjX6KcSIf4XAgPS1yLAwD/bBV4sMxaiSpG9fjWbESo7AQ5/TpOKZO6ZVkoWpfNXetupNqXxVew4smNN458jb3z/oJk5Inc/PcEazeW0xDE7OL26Fw7axhvVoeXtUU5tw2g0/+sR494OsxFRO/w8/ByXuaHSuEwvGaXEbFh6+PQ/bCUWx/dXfzSiACNKfargPdpmOYpsm2bdvYs2cPfr+fzMxMZs6cSXR0N3NnbABbaXQYaZr4Nm1C1tTinJGDEtvRRjxdmEtKvJ9vR4aI7cbpxDVrVuNL4XIRccnSXpPlJC/te4EKbzl64G6nSx3d0Pnjlof4z4VPkJUczT9vmckjK/axM89q93rjnCwun9F7juiTjDl/BDGp0Wx/dTe1JXUcTT7A+uGf4I1srnANUyfOGd/r8rRFTHIUF/5wHh8+9CmG30CaksjESBb/eH6bvhCbjvPBBx+Qm5uLEYgyPHToEHl5eVx77bW4OpBQa9M2ttLoAP59+yj94g3I2hoQCtLvJ+6B+4m+6cu9Ml/tPx+l9qE/Bu9QFERkJNG33dor87bFZ/mfNSqMplT7qimuL2Zw1GDGDInlkZvCE0Y4ZHwKQ8ZbyYpbi7ewfv1qaBJkpgmN7MSxJEcmh0W+pgydmsaNT1xJ+bFKVIdKfEbsgCsl0V+pqqpqpjDAegjz+/3s2bOHKVOmhE+40wT70aYdpGFQet0XMQsLkbV11tO/x0P1z3+BLxB33dPz1fzxT8FlyQERF0fK8ndRU3q2km5HiNBa9oSwkFLi0vrX09vUlGl8ZcLNuFU3kVokTsXJ2KRx/GDmj8ItWiOKqjBoRCIJQ+NshdFBZK0Ps7Q+ZFLlScrKykImzBmGQVFRUW+Kd8ZgrzTawbd+Q8jSItLrpe6pZ3D28JOLWVWF9LbirDV0tIyM0Pt6mSUjlvLYzv80cyQrQmVMwhjiXfFhkaktloy4hIXDLiS3Opd4VxzJbZRMsenfyDofvpd3I49XgSLAoaItGYM2LnjVGBMTQ6jcM0VRiI+P7wNpT39spdEOZk1weXBrh4lZUd7j8ymxsQiXC9kiagpAGx6+qqGLsi5iX/k+1pz4BFVRkVKSFDGI78/4QbfH9voNVu8tJr+igbFpscwYkYTSAw2sXKqL0Qnhc3rb9Ay+53cgC2uthvaGBL+J/uoelAQ3Smrz3KNBgwaRkJBAWVkZpnkqDFtRFCZMCHc14NMDW2m0g3PGzOCyIoCIjMR90UU9Pp/QNGLu+A41Dz3cvCy5203sD+7t8fk6iiIUvjv9e1w39noOVBwgKWIQ4xLHddu0cqK8nq/+Zz0en4HXb+B0qGQmRfH3m2cQ2YtRVzYDA7O4DllSZymMphgm+ro8nMual1YRQnDxxRfz8ccfk5ubC0BcXBzz5s2zo6d6CPtb2Q5qYgKx37+Hmj88ZIW/SomIjEDLHkPksst6Zc7ob3wd4XZR86dHMEtLUYcPJ+6B+3HPndsr83WG1KghpEb1XMexn76yg8o6X+M9ocFncLi4lsc+OsS3F2X32Dw2AxNZ4yVkApAEWRmiDS/gcrm48MIL0XUdwzDsiKkexlYaHSDmG1/HOX0adU8+hVlZRcSlS4lctgzh7J0sZyEE0bfcQvQttyClbHyaL63x8uaWPAoqG5g2PJHzx6c26wE+0Kj1+Nl9oiroIdJvmLz7eb6tNGxQUqMtk1RLNIEyIqHNczVNC6oua9N97He0g7hmzsQ1c2afz3tSYWzPreC7T23GkBJfoKbTk6sP8++vzurV5Llw0RNlNE1psrV4CztKPifeFc+8oQtIcLd9o7HpX4goJ+rMdIyNJ+BkqRhFgNuBNqMHOiTadJrT725zGiKl5Cf/+5wG/6nY8wafwYmKBp799Ai3nz8wnb3RbgfZQ2LZnV/VrMy4QxUsOqt7JjC/6efBtQ9woGI/HsODQ3Hy7N5nGjPYbQYO2gUjUFKj0dflIRv8qKOT0M4dhojo/5ViT0cGrm3jDOJERQOV9cHRVD7dZMWOghBnDBx+csVZxEU4iHBYlYQjnSpDk6K4bf6obo278uiKZq1u/aYPr+Hltxv/D0Ma7Zxt058QQqBOHIzrtum4vzMLx+LRiOjeMQ3btI+90hgAOFUlyO7fuE8beO0im5I5KIpX75rLh7uLyK9oIHtILHPGJKN2M+R21fEPQxYn9Js+DlcetkNxbWy6iK00BgApcW6GJ0exv7CmmRnH7VC4PCc8yX49gWFKnvrkMC+tz6XW42diRjzzxqZ0W2EAqEpoZSqlRBX2AtvGpqvY354wYxQXU/WrX1Ny2TLKv3c3/n37Qh73q2umMCjaRaRLxe1QcWkKs0cnc8WMzD6WuOf43Vu7efKTw1TU+fAbkq3HKvjaYxs4XlbX7bEvHLYIlxpc+iTaGc3wuPAlSdrYDHTslUYY0Y8fp3jxxVaZEp8Ptm7D88abJD7276CcjPTESF69ay4bDpVRXO1h4tB4Rg3uvU58vU1FnY93tufja9Fxz+c3eHrNEX50Wff6e88fuoDNRZvYULgeU5poioaCwo/Pvt+u9WRj0w1spRFGqn/9f8jqajhZ7sAwkA0NVH7/PgavWxt0c9NUhXPG9EyVVrOmhurf/Z6G114HIYi4fBmxd38PJaZvFNHxsjqcqhKkNAwJe/NbKd3SCRSh8P0Z93Go8hA7S3cQ54pj9pDZuFopvGgTKMlfXY4WEYXmtN8nm9CERWkIIa4GHgTGATOllJua7PshcCtWYes7pJTLA9unA08AEcA7wJ0yVGWyAYT3k09OKYwmGMXFmGVlqIMG9cq80jAoWXY5+uEjjZ0B6558Cu+na0l5750u90TvDOkJkfiM4GtXBIxM6blyDyPjRzIyfmSPjXe6cnzDSjY/9gu8tZWAIOvcpUy/5f/ZysMmiHD5NHYCVwCrm24UQowHrgMmAIuBvwkhTt7B/g7cDowO/CzuM2l7CdFGIycRGdlr83o++BDjeF7zVrI+H8axY3g/+rjX5m1KUoyL+eNScLXIaHdqCjeeZ/sc+pKSfVv57JF7aagoxvT7MP1ejq55i/V/6z+l5G36D2FRGlLKPVLKUB7fy4AXpJReKeUR4CAwUwgxBIiVUn4WWF08BSzrO4l7h+iv3oaIaNEX2ukkYtGFKL2oNPy7diHrgp3NsqEB/65dvTZvS+5fdhZXzBiK26EgsFYYf7wxhxE9uNKwaZ/drz2K4Wtex8n0e8nb9AGe6p6v5GwzsOlvPo10YF2T13mBbf7A3y23h0QIcTvWqoTMzP4bXRT15RvR9++n7oUXEU4n0u/HOX0a8b/7ba/Oqw0dioiKClIcIiICNbP327OexKEp3Ll4LHcsysYwJZpqB/OFg5rC3JDbFc1JQ3kR7tjEPpbIpj/Ta99SIcT7QoidIX7aKg0bKqxFtrE9JFLKR6WUOVLKnOTk8Lf3bA2hKMT/6pekrv+MxH89yuD3V5D80ou97oyOWHIxwu2Gph3OAq1kIxb3vdVPCGErjDCSnD0FESKvRRo60an996HLJjz02kpDSrmwC6flAU0fdTOA/MD2jBDbTwvU5GTUPlRuIiKC5Ddeo+Ku7+HbshUA5/RpJPzxYUuZ2Aww6oETQDIQ3+mzxy/7GrmfLUf31nMye1R1RTBu6c043FE9KajNaUB/M0+9ATwnhHgISMNyeG+QUhpCiBohxCxgPfBl4JEwyjng0bKySH71FczaWhACJcq+OQw8JPAolotPw7LiXgDcD3S8NlNMaiYX/vJFtj//MCV7N+OKTWT8ZbcxfN6yXpDZZqATrpDby7Fu+snA20KIbVLKRVLKXUKIl4DdgA58S8rG6nLf4FTI7buBH5tuotjdzAYwbwBPA97AD8CHWF+RzkU+xWWMZO73/9Kj0tmcnogBnurQLjk5OXLTpk3tH3iGYpRXUPPXv+JZvgIlLo7o224hYtkyO2t6QHAFEMqJ7QQ+ojOrDRublgghNkspc1pu72/mKZs+xKyupmTRYozSEvD5MYDKe+/D9/kO4n/yQLjFs2mXila2Syw/h600bHoeO2TlDKbuuecxy8vB52/cJusbqHvyKYzi4jBKZtMxJhE6sDABiOtjWWzOFGylcQbj/fhjpMcTtF04nfi2fx4GiWw6x3ew/BdNw2XdwL2EViY2Nt3HNk+dwagZQ0FVwWjRyc4wUFMHh0eogYyU0Ke+oFFYjvDHgB1Y0eo3A1P6UAabMw1baZzBRN96Mw2vvopsaDi1UVVRM4fimNi90uRnFCc2wTvfgvxN4IiGGV+HBT8HrS98CsOAn/bBPDY2FrZ56gzGMXYsCX99BCUxwSqQ6HLhnDqFQc89a0dPdZSyA/DEAjixAaQJvmpY/wi89pVwS2Zj0yvYK40znIhFi3AvXIh++DBKdDTqkCHhFmlgsfb3oLfwC+kNsOdVqM6H2LTwyGVj00vYSsMGoao4Ro8OtxgDk4KtIPXg7ZoLyg/YSsPmtMM2T9nYdIch00CEePYyvJBoK2Kb0w9badjYdIdz7gFHiyKPWgSMu8JeZdiclthKw8amOySNgq98DBmzQajgjodZd8GyJ8ItmY1Nr2D7NGxsukvaNLhtbbilsLHpE2ylYWPTgtLD5RTvKyVqUCRDp6Wh2A2ibGwasZWGjU0AQzdZ8euPyd9RiJSgqAJnpINLf72I2MF2CXkbG7B9GjY2jex8ay8nPi9E9xoYPgN/g059hYf3f/tJuEWzsek32ErDxibA3hUHMXzN63BJU1J+rIK68vowSWVj07+wlYaNTQBDN0LvEAJTN/tWGBubfoqtNGxsAow8NwvVEfyViEqMIDrZ7qFuYwO20rCxaWTqVROITY1Bc1vxIapTxRGhcf73zrULONrYBLCjp2xsAjgjnVz58MUc+ew4BbuLiEmJJvv8kUTEu9s/2cbmDMFWGjY2TVAdKqPmZjFqbla4RbGx6ZfY5imbfo3f8LO3fA+51ceQUoZbHBubM56wrDSEEL8DLgF8wCHgZillZWDfD4FbAQO4Q0q5PLB9OvAEVlPkd4A7pX0XOa1Znfcxf932CAKBIQ2SI5K5f/aDDImye37Y2ISLcK00VgITpZSTgP3ADwGEEOOB64AJwGLgb0IINXDO34HbgdGBn8V9LbRN33Gk6gh/3vonGvQG6vV6vIaXE7UnuH/NjzClHf5qYxMuwqI0pJQrpGzsXLMOyAj8fRnwgpTSK6U8AhwEZgohhgCxUsrPAquLp4BlfS23Td/x7pG30U1/s20SSY2/hj1lu8MklY2NTX/wadwCvBv4Ox043mRfXmBbeuDvltttTlPKPeWtrCgE1b7qPpfHxsbGoteUhhDifSHEzhA/lzU55seADjx7clOIoWQb21ub+3YhxCYhxKaSkpLuXIZNmMgZPAOX6grarpt+xiaODYNENjY20IuOcCnlwrb2CyFuApYCFzRxaOcBQ5sclgHkB7ZnhNje2tyPAo8C5OTk2M7yAciCzPN56/AbFNYV4jN9ALhUF5eMvIwEd2KYpbOxOXMJV/TUYuA+YJ6UsmkluDeA54QQDwFpWA7vDVJKQwhRI4SYBawHvgw80tdy24Rmy9FyHvvoEHnl9YxLj+O2+SMZOTimW2O6VBe/m/cQ7x55h09PfEKUI5qlIy5h5pCze0hqGxubriDCEbUqhDgIuICywKZ1UsqvB/b9GMvPoQPflVK+G9iew6mQ23eB73Qk5DYnJ0du2rSpx6/BxmLV7kIefGUHXr/lfxACXJrKP2+dSfaQ2DBLZ2Nj01WEEJullDlB20/3VAdbafQeUkou/cPHlNR4g/bNHJnEn78c9P9mY2MzQGhNafSH6CmbAUqNR6ei3hdy3668qj6WxsbGpi+wlYZNl4l0qqitVH9NjHb2sTQ2NjZ9ga00bLqMpiosy8nA1aIHhduhcNN5w8MklY2NTW9iV7kdQJgVFdT972X0AwdwTp1KxLLLUCIiwirTdy7Mxus3eWd7PpoiMCV8Ze5wlkyxcy9tbE5HbEf4AMG/fz8ll10OPh/S40FERiLi40l55y3U5ORwi0edR6e01svgODduh9r+CTY2Nv0a2xE+wKm4+x5kTQ3S4wFA1tdjFhVR9ctfhVkyiyi3xrBBUbbCsLE5zbGVxgDAbGjAv/1zaLkqNAw8y1eERygbG5szEltpDACEolhZc6H2OWy3lI2NTd9hK40BgHC5cM2fD1oLBeFyEXHNNWGRycbG5szEVhoDhITf/xZt2DBEVBRERCAiI3FOnkzs3d8Lt2g2NjZnELZtY4CgJieT8tGHeNd8ipGbi2PcOBzTpiJaMVvZ2NjY9Aa20hhACEXBPfe8cIthY2NzBmObp2xsbGxsOoytNGxsbGxsOoytNGxsbGxsOoytNGxsbGxsOoytNGxsbGxsOsxpX7BQCFECHAu3HF1gEFAabiG6yelwDXB6XId9Df2HgXIdw6SUQdVQT3ulMVARQmwKVWFyIHE6XAOcHtdhX0P/YaBfh22esrGxsbHpMLbSsLGxsbHpMLbS6L88Gm4BeoDT4Rrg9LgO+xr6DwP6Omyfho2NjY1Nh7FXGjY2NjY2HcZWGjY2NjY2HcZWGv0MIcRiIcQ+IcRBIcQPwi1PVxBCDBVCrBJC7BFC7BJC3BlumbqKEEIVQmwVQrwVblm6ihAiXgjxPyHE3sBnMjvcMnUWIcRdgf+lnUKI54UQ7nDL1B5CiMeEEMVCiJ1NtiUKIVYKIQ4EfieEU8auYCuNfoQQQgX+ClwEjAeuF0KMD69UXUIH7pZSjgNmAd8aoNcBcCewJ9xCdJM/Ae9JKccCkxlg1yOESAfuAHKklBMBFbguvFJ1iCeAxS22/QD4QEo5Gvgg8HpAYSuN/sVM4KCU8rCU0ge8AFwWZpk6jZSyQEq5JfB3DdZNKj28UnUeIUQGsAT4d7hl6SpCiFhgLvAfACmlT0pZGVahuoYGRAghNCASyA+zPO0ipVwNlLfYfBnwZODvJ4FlfSlTT2Arjf5FOnC8yes8BuDNtilCiCxgKrA+zKJ0hT8C9wJmmOXoDiOAEuDxgJnt30KIqHAL1RmklCeA3wO5QAFQJaVcEV6pusxgKWUBWA9XQEqY5ek0ttLoX4Tq3TpgY6KFENHAy8B3pZTV4ZanMwghlgLFUsrN4Zalm2jANODvUsqpQB0DzCQSsPtfBgwH0oAoIcQN4ZXqzMVWGv2LPGBok9cZDIBleCiEEA4shfGslPKVcMvTBeYAlwohjmKZCc8XQjwTXpG6RB6QJ6U8udL7H5YSGUgsBI5IKUuklH7gFeCcMMvUVYqEEEMAAr+LwyxPp7GVRv9iIzBaCDFcCOHEcva9EWaZOo0QQmDZ0PdIKR8KtzxdQUr5QyllhpQyC+tz+FBKOeCebqWUhcBxIUR2YNMFwO4witQVcoFZQojIwP/WBQwwZ34T3gBuCvx9E/B6GGXpElq4BbA5hZRSF0J8G1iOFSHymPz/7d0/SJVRHMbx70MJRQYVtkUEjjkY6lAt1WBbCNkUoSENDW65FBH0B4rG9nCJoqyGIrC72CKEWVoJSRQNQpsNFS3Gr+E9F283e33v1bDs+cAF33vfc8/Zfp7j6/OLmFrhZdVjL3AMeCVpIr13OiIerdyS/mv9wI30i8h74PgKr6cmEfFU0hDwnOzJvBf8A1Eckm4C+4AmSTPAOeAycFtSH1kxPLJyK6yPY0TMzKwwH0+ZmVlhLhpmZlaYi4aZmRXmomFmZoW5aJiZWWEuGmY5JH2XNFHx2lHHd3T9ycBGST0pNfWtpJ7FR5jVz4/cmuWQ9CUiGpf4HYPAw4gYqmHM2oiYK3DfFuAZ0E4WOTMOtEXEpzqXa5bLOw2zGklqk/RE0rik4YpYiBOSxiRNSrqb/oN5D3AIuJp2Ks2SRiS1pzFNKaoESb2S7kh6ADyWtCH1ZBhLYYMLJR4fBEoRMZsKRYlf47jNlo2Lhlm+9RVHU/dTptY1oDsi2oDrwKV0772I6IiIcs+KvogYJYuOGIiI1oh4t8h8u4GeiDgAnCGLL+kA9pMVnuqE2lWXjGx/N8eImOX7FhGt5QtJLUALUMpikFhDFtcN0CLpIrAJaCSLg6lVKSLKPRg6yUITT6XrdcB2fs5dWlXJyPb3c9Ewq42AqYhYqGXqINAVEZOSeslyhxYyx/wuv7pt6dequQ5HxHTOemaq5tkGjOTcb7YkPp4yq800sLXcZ1tSg6Sd6bONwMd0hHW0Yszn9FnZB6At/dydM9cw0J+SXZG06zf3dEranPpOdFLfDsesEBcNsxqkNrzdwBVJk8AE870dzpJ1KCwBbyqG3QIG0h+zm8m60J2UNAo05Ux3AWgAXkp6na6r1zOb3h9Lr/MVx1tmy86P3JqZWWHeaZiZWWEuGmZmVpiLhpmZFeaiYWZmhblomJlZYS4aZmZWmIuGmZkV9gMTfuLtDVeoNAAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"fig, ax = plt.subplots()\n",
"ax.scatter(X2[::5, 0], y2[::5], c=np.arange(0, len(X2), 5) // 100, cmap=\"Set1\",\n",
" label=\"Partition\")\n",
"ax.set(xlabel=\"Feature 0\", ylabel=\"target\", title=\"Non-stationary data (by partition)\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's fit two estimators:\n",
"\n",
"1. One `BlockwiseVotingRegressor` on the entire dataset (which fits a `LinearRegression` on each partition)\n",
"2. One `LinearRegression` on a sample from the entire dataset"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:56.339454Z",
"iopub.status.busy": "2021-10-14T16:48:56.338834Z",
"iopub.status.idle": "2021-10-14T16:48:56.513378Z",
"shell.execute_reply": "2021-10-14T16:48:56.513899Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LinearRegression()\n",
"clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n",
" subestimator,\n",
")\n",
"clf.fit(X2, y2)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:56.527589Z",
"iopub.status.busy": "2021-10-14T16:48:56.527083Z",
"iopub.status.idle": "2021-10-14T16:48:56.638812Z",
"shell.execute_reply": "2021-10-14T16:48:56.639884Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"LinearRegression()"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_sampled, y_sampled = dask.compute(X2[::10], y2[::10])\n",
"\n",
"subestimator.fit(X_sampled, y_sampled)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Comparing the scores, we find that the sampled dataset performs much better, despite training on less data."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:56.643349Z",
"iopub.status.busy": "2021-10-14T16:48:56.642549Z",
"iopub.status.idle": "2021-10-14T16:48:57.053080Z",
"shell.execute_reply": "2021-10-14T16:48:57.052462Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"-11.43572921540219"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.score(X2, y2)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"execution": {
"iopub.execute_input": "2021-10-14T16:48:57.060821Z",
"iopub.status.busy": "2021-10-14T16:48:57.060022Z",
"iopub.status.idle": "2021-10-14T16:48:57.222201Z",
"shell.execute_reply": "2021-10-14T16:48:57.220599Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.14965772401417687"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"subestimator.score(X2, y2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This shows that ensuring your needs to be relatively uniform across partitions. Even including the standard controls to normalize whatever underlying force is generating the non-stationary data (e.g. a time trend compontent or differencing timeseries data, dummy variables for geographic regions, etc) is not sufficient when your dataset is partioned by the non-uniform variable. You would still need to either shuffle your data prior to fitting, or just sample and fit the sub-estimator on the sub-sample that fits in memory."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}