{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Blockwise Ensemble Methods\n",
"\n",
"Dask-ML provides some [ensemble methods](https://ml.dask.org/modules/api.html#module-dask_ml.ensemble) that are tailored to `dask.array`'s and `dask.dataframe`'s blocked structure. The basic idea is to fit a copy of some sub-estimator to each block (or partition) of the dask Array or DataFrame. Becuase each block fits in memory, the sub-estimator only needs to handle in-memory data structures like a NumPy array or pandas DataFrame. It also will be relatively fast, since each block fits in memory and we won't need to move large amounts of data between workers on a cluster. We end up with an ensemble of models: one per block in the training dataset.\n",
"\n",
"At prediction time, we combine the results from all the models in the ensemble. For regression problems, this means averaging the predictions from each sub-estimator. For classification problems, each sub-estimator votes and the results are combined. See https://scikit-learn.org/stable/modules/ensemble.html#voting-classifier for details on how they can be combeind. See https://scikit-learn.org/stable/modules/ensemble.html for a general overview of why averaging ensemble methods can be useful.\n",
"\n",
"It's crucially important that the distribution of values in your dataset be relatively uniform across partitions. Otherwise the parameters learned on any given partition of the data will be poor for the dataset as a whole. This will be shown in detail later."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's randomly generate an example dataset. In practice, you would load the data from storage. We'll create a `dask.array` with 10 blocks."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:55:55.495803Z",
"iopub.status.busy": "2021-01-14T10:55:55.495239Z",
"iopub.status.idle": "2021-01-14T10:55:58.941855Z",
"shell.execute_reply": "2021-01-14T10:55:58.942553Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
160.00 MB
16.00 MB
\n",
"
Shape
(1000000, 20)
(100000, 20)
\n",
"
Count
10 Tasks
10 Chunks
\n",
"
Type
float64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from distributed import Client\n",
"import dask_ml.datasets\n",
"import dask_ml.ensemble\n",
"\n",
"client = Client(n_workers=4, threads_per_worker=1)\n",
"\n",
"X, y = dask_ml.datasets.make_classification(n_samples=1_000_000,\n",
" n_informative=10,\n",
" shift=2, scale=2,\n",
" chunks=100_000)\n",
"X"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Classification\n",
"\n",
"The `sub-estimator` should be an instantiated scikit-learn-API compatible estimator (anything that implements the `fit` / `predict` API, including pipelines). It only needs to handle in-memory datasets. We'll use `sklearn.linear_model.RidgeClassifier`.\n",
"\n",
"To get the output shapes right, we require that you provide the `classes` for classification problems, either when creating the estimator or in `.fit` if the sub-estimator also requires the classes."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:55:58.946806Z",
"iopub.status.busy": "2021-01-14T10:55:58.946388Z",
"iopub.status.idle": "2021-01-14T10:55:58.951014Z",
"shell.execute_reply": "2021-01-14T10:55:58.950676Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"BlockwiseVotingClassifier(classes=[0, 1],\n",
" estimator=RidgeClassifier(random_state=0))"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sklearn.linear_model\n",
"\n",
"subestimator = sklearn.linear_model.RidgeClassifier(random_state=0)\n",
"clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n",
" subestimator,\n",
" classes=[0, 1]\n",
")\n",
"clf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can train normally. This will *independently* fit a clone of `subestimator` on each partition of `X` and `y`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:55:58.963174Z",
"iopub.status.busy": "2021-01-14T10:55:58.952720Z",
"iopub.status.idle": "2021-01-14T10:56:01.693477Z",
"shell.execute_reply": "2021-01-14T10:56:01.692668Z"
}
},
"outputs": [],
"source": [
"clf.fit(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of the fitted estimators are available at `.estimators_`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:01.698328Z",
"iopub.status.busy": "2021-01-14T10:56:01.697925Z",
"iopub.status.idle": "2021-01-14T10:56:01.701943Z",
"shell.execute_reply": "2021-01-14T10:56:01.702543Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0)]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.estimators_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These are different estimators! They've been trained on separate batches of data and have learned different parameters. We can plot the difference in the learned `coef_` of the first two models to visualize this."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:01.705196Z",
"iopub.status.busy": "2021-01-14T10:56:01.704376Z",
"iopub.status.idle": "2021-01-14T10:56:01.950499Z",
"shell.execute_reply": "2021-01-14T10:56:01.950080Z"
}
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:01.968389Z",
"iopub.status.busy": "2021-01-14T10:56:01.967230Z",
"iopub.status.idle": "2021-01-14T10:56:02.066594Z",
"shell.execute_reply": "2021-01-14T10:56:02.066918Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEFCAYAAAAWrxseAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVNUlEQVR4nO3df7BkZX3n8feHAcRfEQij/ByG4MRywI2rI6C7qSUqyo/gUCtWQe0uSFRWS5KqLTWMy7rZZHGDawxKpGTZLIKaWopsljgKLiKG1KpFwrCKZlTCiPwYGWEgiCIagnz3j/NMbC793Nv39p2fvF9VXfec8zzPOc/p7tuffs453Z2qQpKkcXbb3h2QJO24DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEk9jSS5J8r6R+XckuS/JI0l+Mck/S3J7mz9lO3Z1wZKsT3Ls9u7HtpLkxiRv3Q7bTZKPJ3koyV+3ZTOfT48k+aU51rOs1VuybXquucTPSeyaktwJvAB4HPgZ8E3gE8ClVfXEmPp7AD8EjqmqW9uyG4C1VfWRbdXvHUmSAlZU1Ybt3ZdJJbkR+FRV/XGn/JeB9wO/BuwB3AVcDnykqn42xXZ/FfifwIuq6sfjnk/b2lz3hSbjSGLXdnJVPRc4FLgAOBf4H526LwD2AtaPLDt0xvzEkuy+kHZPJ+3d9zb7H0xyOPBXwD3AS6rqecCbgFXAc6dc/aHAnVX14zY/7vmknVFVedsFb8CdwGtnLDsKeAI4ss1fDpwP/DLwY6CAR4AvAt9pdX/Slj0DeB5DyGwCvtfaLmnrejPwZeBC4O9a2TOAPwDuBu4DLgGe2eofC2wE3gXc39Z51khfnwl8iOGd7sPAl0baHgN8BfgBcCtw7CT3A/CfgKsYRlQ/YngBWzVL2wJeOGb5bPu1D/BZYDPwUJs+eKTtjQzv5L/c7tsXtu28Hbi9tbmYNspvbX4D+FYruw44dKTsOODb7T76KPCXwFs7+/Mp4Jo5njdvaPfLD1pfXzxSdiDwZ23fvgv8Vlv+FuCnDCPWRxhGFE96Ps28P3uPL7C81du91ZvrOfel9lg81Pp0Qit7f+vPT1sfPgqE4fl5f9vm12n/C95meU5s7w5420oP7JiQaMvvBt7Rpi8Hzm/TT/rnHLcO4M+B/wY8G3g+8NfAv21lb2Y4tPWbwO7tH/7DwFpgX4Z3qp8Bfr/VP7bV/z2Gwx4nAo8C+7Tyi9uL1EHAEuBVDC/OBwEPtvq7MbxIPggsnet+YAiJn7a2S4DfB26a5T7shcRs+/WLwBuBZ7WyPwX+fKTtje0xOKLdT3u07XwW2BtYxvAifHyrfwqwAXhxq/8fgK+0sv0YDumc2tbz79p92guJ7zMSxGPKt7xZOK6t77fbtvds9/UtwH9s878E3AG8fuTx/9LIupbz1OfTaEj0Ht8ntWPu59w/AG9r63gHcC8/P4x+4+h9Aby+7cPeDIHxYuCA7f2/uqPftnsHvG2lB7YfEjcB57Xpy5kwJBgOH/w97R1zW3Y68Bdt+s3A3SNlaS84h48seyXw3TZ9LMM76dHt3c8wStitlf3KmP6fC3xyxrLrgDPnuh8YQuILI2UrgZ/Mch8+JSTm2q8x63gp8NDI/I3A743Zzj8fmb8KWNOmPwe8ZaRsN4YwPRQ4g5GQa33bSD8k/oEWPp3y9wFXzdjW99pjdfTo49vK3wt8fOTxnygk5nh8/7HdhM+5DSNlz2pt9x+5r0dD4tXA3255jm3t/8Fd5eZx46efgxgOB83XoQzvLjcl2bJsN4bj21uMTi9l+Ke9ZaR+GN7xbfFgVT0+Mv8o8ByGd8h7MRzyGtePNyU5eWTZHsBfTLgf35+xvb2S7D6jH7OZdb+SPIvhkMbxDIeeAJ6bZEn9/MTw6P3U69dz2vShwEeSfGikPAyP44Gj66qqSjJu3Vs8CBwwS/mBDId/tqzviba+gxgC5sAkPxipvwT4v7Osr2e2x3fUJM+5f7zfqurRVu85jFFVX0zyUYZRzLIkVwPvrqofLmAfnjYMiaeRJK9g+If/0gKa38Pwrm6/WV5Qa2T6AYZ3i0dU1ffmua0HGA4LHc5wzmFmPz5ZVW+b5zoXy1z79S7gRcDRVfX9JC8Fvsrwwr5FjWnXcw/w/qr6k5kFSVYAh4zMZ3R+jC8wHAr7eKf8XuAlY9b3PYbH/rtVtWIefe+Z7fEdNclzbjZPuZ+r6iLgoiTPZxixvYdhBKUOr256GkjyC0l+HbiS4ZLAb8x3HVW1Cfg88KG2vt2SHJ7kX3TqPwH8d+DC9g9JkoOSvH6CbT0BXAb8YZIDkyxJ8sokz2A4+Xpykte35XslOTbJwfPdpwnt2baxV5K9GF7sZ9uv5zKEyA+S7Av8zpTbvwR4b5Ij2rael+RNrewa4Igk/7JdTfZbwP6zrOt3gFcl+WCS/dv6XpjkU0n2ZnjRPCnJa9olrO9ieJH+CsO5gB8mOTfJM9t9f2R74zEvczy+o/Xm9Zwb4z6Gcye0fX1FkqPbvv2Yn59s1ywMiV3bZ5L8iOEd2XnAHwJnTbG+MxhOWn6T4WqS/8Xshy/OZTjxeVOSHzK8k33RhNt6N/AN4GaGw2MfYDiOfA+wGvj3DCd472F4N7i1nsvrGV70t9zOYvb9+jDDSfsHGM7//J9pNl5VVzPs+5VtW38DnNDKHmC4hPUChkNJKxiumuqt6zsM50+WA+uTPMxwtdI64EdVdRvwr4E/av0/meEy6sfaobKTGc6xfLeV/zHD1UcLMfbxHVNvvs+5UR8BTm0f8LsI+AWGgH+I4bDagwxXRmkWfphOktTlSEKS1GVISJK6DAlJUpchIUnq2qU+J7HffvvV8uXLt3c3JGmncssttzxQVUvHle1SIbF8+XLWrVu3vbshSTuVJHf1yjzcJEnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVLXLvVhOknaUSxfc82829x5wUlboSfTWZSRRJLjk9yWZEOSNWPKk+SiVv71JC+bR9t3J6kk+y1GXyVJk5s6JJIsYfhh8ROAlcDpSVbOqHYCw69mrQDOBj42SdskhwDHAXdP209J0vwtxkjiKGBDVd1RVY8x/I7y6hl1VgOfqMFNwN5JDpig7YXAbzO/H46XJC2SxQiJgxh+Z3iLjW3ZJHW6bZO8AfheVd26CH2UJC3AYpy4zphlM9/59+qMXZ7kWcB5wOvm3HhyNsMhLJYtWzZXdUnSPCzGSGIjcMjI/MHAvRPW6S0/HDgMuDXJnW35/0uy/8yNV9WlVbWqqlYtXTr269AlSQu0GCFxM7AiyWFJ9gROA9bOqLMWOKNd5XQM8HBVbeq1rapvVNXzq2p5VS1nCJOXVdX3F6G/kqQJTX24qaoeT3IOcB2wBLisqtYneXsrvwS4FjgR2AA8Cpw1W9tp+yRJWhyL8mG6qrqWIQhGl10yMl3AOydtO6bO8ul7KUmaL7+WQ5LUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuf5lO0g5tvr/wtiP+utvOzJGEJKnLkJAkdRkSkqQuQ0KS1GVISJK6vLppBzDfqzfAKzgkbRuOJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYsSEkmOT3Jbkg1J1owpT5KLWvnXk7xsrrZJPpjk263+1Un2Xoy+SpImN3VIJFkCXAycAKwETk+ycka1E4AV7XY28LEJ2l4PHFlV/wT4W+C90/ZVkjQ/izGSOArYUFV3VNVjwJXA6hl1VgOfqMFNwN5JDpitbVV9vqoeb+1vAg5ehL5KkuZhMULiIOCekfmNbdkkdSZpC/AbwOfGbTzJ2UnWJVm3efPmeXZdkjSbxQiJjFlWE9aZs22S84DHgT8Zt/GqurSqVlXVqqVLl07QXUnSpHZfhHVsBA4ZmT8YuHfCOnvO1jbJmcCvA6+pqpnBI0nayhZjJHEzsCLJYUn2BE4D1s6osxY4o13ldAzwcFVtmq1tkuOBc4E3VNWji9BPSdI8TT2SqKrHk5wDXAcsAS6rqvVJ3t7KLwGuBU4ENgCPAmfN1rat+qPAM4DrkwDcVFVvn7a/W8vyNdfMq/6dF5y0S2xb0q5tMQ43UVXXMgTB6LJLRqYLeOekbdvyFy5G3yRJC+cnriVJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkrkX5gj/p6cRv3dXTiSMJSVKXISFJ6jIkJEldhoQkqcuQkCR1eXXTCK9akaQncyQhSeoyJCRJXYaEJKnLkJAkdXniWgs23xP94Ml+aWfjSEKS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLj9M9zTnN99Kmo0jCUlSlyEhSerycJO2Gw91STs+RxKSpC5DQpLUZUhIkroWJSSSHJ/ktiQbkqwZU54kF7Xyryd52Vxtk+yb5Pokt7e/+yxGXyVJk5s6JJIsAS4GTgBWAqcnWTmj2gnAinY7G/jYBG3XADdU1QrghjYvSdqGFmMkcRSwoaruqKrHgCuB1TPqrAY+UYObgL2THDBH29XAFW36CuCUReirJGkeUlXTrSA5FTi+qt7a5v8NcHRVnTNS57PABVX1pTZ/A3AusLzXNskPqmrvkXU8VFVPOeSU5GyG0QnLli17+V133TXV/mjnMO3ls9vr8ttpf/J1mn5vy7aLue1p7Kz93tbbTnJLVa0aV7YYI4mMWTYzeXp1Jmk7q6q6tKpWVdWqpUuXzqepJGkOixESG4FDRuYPBu6dsM5sbe9rh6Rof+9fhL5KkuZhMULiZmBFksOS7AmcBqydUWctcEa7yukY4OGq2jRH27XAmW36TODTi9BXSdI8TP21HFX1eJJzgOuAJcBlVbU+ydtb+SXAtcCJwAbgUeCs2dq2VV8AXJXkLcDdwJum7askaX4W5bubqupahiAYXXbJyHQB75y0bVv+IPCaxeifJGlh/MS1JKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnq8jeuJanD31V3JCFJmoUhIUnqMiQkSV2GhCSpyxPXkrQD2lFOmhsS2intKP9A0q7OkJC0y/LNxPQ8JyFJ6jIkJEldhoQkqcuQkCR1GRKSpC6vbpJ2Il6to23NkYQkqcuQkCR1GRKSpC7PSUja6jyXsvNyJCFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlx+mk7YhP1SmnY0jCUlSlyEhSeoyJCRJXYaEJKnLkJAkdU0VEkn2TXJ9ktvb33069Y5PcluSDUnWzNU+yXFJbknyjfb31dP0U5K0MNOOJNYAN1TVCuCGNv8kSZYAFwMnACuB05OsnKP9A8DJVfUS4Ezgk1P2U5K0ANOGxGrgijZ9BXDKmDpHARuq6o6qegy4srXrtq+qr1bVvW35emCvJM+Ysq+SpHmaNiReUFWbANrf54+pcxBwz8j8xrZs0vZvBL5aVX8/rgNJzk6yLsm6zZs3L3A3JEnjzPmJ6yRfAPYfU3TehNvImGU1UcPkCOADwOt6darqUuBSgFWrVk20XknSZOYMiap6ba8syX1JDqiqTUkOAO4fU20jcMjI/MHAlkNJ3fZJDgauBs6oqu9MsC+SpEU27eGmtQwnlml/Pz2mzs3AiiSHJdkTOK2167ZPsjdwDfDeqvrylH2UJC3QtCFxAXBcktuB49o8SQ5Mci1AVT0OnANcB3wLuKqq1s/WvtV/IfC+JF9rt3HnKyRJW9FU3wJbVQ8Crxmz/F7gxJH5a4Fr59H+fOD8afomSZqen7iWJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqWuqD9NJ2nncecFJ27sL2gk5kpAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKX3wIraU5+g+zTlyMJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6pgqJJPsmuT7J7e3vPp16xye5LcmGJGsmbZ9kWZJHkrx7mn5KkhZm2pHEGuCGqloB3NDmnyTJEuBi4ARgJXB6kpUTtr8Q+NyUfZQkLdC0IbEauKJNXwGcMqbOUcCGqrqjqh4DrmztZm2f5BTgDmD9lH2UJC3QtCHxgqraBND+Pn9MnYOAe0bmN7Zl3fZJng2cC/zuXB1IcnaSdUnWbd68ecE7Ikl6qt3nqpDkC8D+Y4rOm3AbGbOs5mjzu8CFVfVIMq75yIqqLgUuBVi1atVc65UAuPOCk7Z3F6SdwpwhUVWv7ZUluS/JAVW1KckBwP1jqm0EDhmZPxi4t0332h8NnJrkvwJ7A08k+WlVfXTuXZIkLZZpDzetBc5s02cCnx5T52ZgRZLDkuwJnNbaddtX1a9W1fKqWg58GPgvBoQkbXvThsQFwHFJbgeOa/MkOTDJtQBV9ThwDnAd8C3gqqpaP1t7SdKOIVW7zmH8VatW1bp167Z3NyRpp5LklqpaNa7MT1xLkroMCUlSlyEhSeoyJCRJXYaEJKlrl7q6Kclm4K7t3Q9J2skcWlVLxxXsUiEhSVpcHm6SJHUZEpKkLkNCktRlSEhzSPKzJF8buS1fwDpOGflFRmmnMedXhUviJ1X10inXcQrwWeCbkzZIsnv7gkxpu3EkIS1Akpcn+csktyS5rv0eCkneluTmJLcm+bMkz0ryKuANwAfbSOTwJDcmWdXa7Jfkzjb95iR/muQzwOeTPDvJZW2dX02yutcnaWswJKS5PXPkUNPVSfYA/gg4tapeDlwGvL/V/d9V9Yqq+hWGr8Z/S1V9heG3U95TVS+tqu/Msb1XAmdW1asZfgHyi1X1CuDXGILm2VthH6WxPNwkze1Jh5uSHAkcCVzffl53CbCpFR+Z5HyGX1R8DsPvqMzX9VX1d236dcAbkry7ze8FLGMIIGmrMySk+QuwvqpeOabscuCUqro1yZuBYzvreJyfj+T3mlH24xnbemNV3bbg3kpT8HCTNH+3AUuTvBIgyR5JjmhlzwU2tUNS/2qkzY9a2RZ3Ai9v06fOsq3rgN9MG7Ik+afTd1+anCEhzVNVPcbwwv6BJLcCXwNe1YrfB/wVcD3w7ZFmVwLvaSefDwf+AHhHkq8A+82yuf8M7AF8PcnftHlpm/G7myRJXY4kJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlS1/8HUH1GyIys1qUAAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"a = clf.estimators_[0].coef_\n",
"b = clf.estimators_[1].coef_\n",
"\n",
"fig, ax = plt.subplots()\n",
"ax.bar(np.arange(a.shape[1]), (a - b).ravel())\n",
"ax.set(xticks=[], xlabel=\"Feature\", title=\"Difference in Learned Coefficients\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That said, the assumption backing this entire process is that the distribution of the data is relatively uniform across partitions. The parameters learned by the each member of the ensemble should be relatively similar, and so will give relatively similar predictions when applied to the same data.\n",
"\n",
"When you `predict`, the result will have the same chunking pattern as the input array you're predicting for (which need not match the partitioning of the training data)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:02.078883Z",
"iopub.status.busy": "2021-01-14T10:56:02.076326Z",
"iopub.status.idle": "2021-01-14T10:56:02.082580Z",
"shell.execute_reply": "2021-01-14T10:56:02.082069Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
8.00 MB
800.00 kB
\n",
"
Shape
(1000000,)
(100000,)
\n",
"
Count
31 Tasks
10 Chunks
\n",
"
Type
int64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array<_vote_block, shape=(1000000,), dtype=int64, chunksize=(100000,), chunktype=numpy.ndarray>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds = clf.predict(X)\n",
"preds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This generates a set of tasks that\n",
"\n",
"1. Calls `subestimator.predict(chunk)` for each subestimator (10 in our case)\n",
"2. Concatenates those predictions together\n",
"3. Somehow averages the predictions to a single overall prediction\n",
"\n",
"We used the default `voting=\"hard\"` strategy, which means we just choose the class that had the higest number of votes. If the first two sub-estimators picked class `0` and the other eight picked class `1` for the first row, the final prediction for that row will be class `1`."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:02.086460Z",
"iopub.status.busy": "2021-01-14T10:56:02.086065Z",
"iopub.status.idle": "2021-01-14T10:56:02.968944Z",
"shell.execute_reply": "2021-01-14T10:56:02.968038Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([0, 1, 0, 0, 0, 1, 0, 1, 1, 0])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds[:10].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With `voting=\"soft\"` we have access to `predict_proba`, as long as the subestimator has a `predict_proba` method. These subestimators should be well-calibrated for the predictions to be meaningful. See [probability calibration](https://scikit-learn.org/stable/modules/calibration.html#calibration) for more."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:02.986496Z",
"iopub.status.busy": "2021-01-14T10:56:02.974718Z",
"iopub.status.idle": "2021-01-14T10:56:05.723858Z",
"shell.execute_reply": "2021-01-14T10:56:05.723041Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LogisticRegression(random_state=0)\n",
"clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n",
" subestimator,\n",
" classes=[0, 1],\n",
" voting=\"soft\"\n",
")\n",
"clf.fit(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:05.741082Z",
"iopub.status.busy": "2021-01-14T10:56:05.734851Z",
"iopub.status.idle": "2021-01-14T10:56:05.897709Z",
"shell.execute_reply": "2021-01-14T10:56:05.897136Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.82142852, 0.17857148],\n",
" [0.12007944, 0.87992056],\n",
" [0.94876877, 0.05123123],\n",
" [0.91949217, 0.08050783],\n",
" [0.56871399, 0.43128601]])"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"proba = clf.predict_proba(X)\n",
"proba[:5].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The stages here are similar to the `voting=\"hard\"` case. Only now instead of taking the majority vote we average the probabilities predicted by each sub-estimator."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regression\n",
"\n",
"Regression is quite similar. The primary difference is that there's no voting; predictions from estimators are always reduced by averaging."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:05.899572Z",
"iopub.status.busy": "2021-01-14T10:56:05.899160Z",
"iopub.status.idle": "2021-01-14T10:56:06.024477Z",
"shell.execute_reply": "2021-01-14T10:56:06.024900Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
160.00 MB
16.00 MB
\n",
"
Shape
(1000000, 20)
(100000, 20)
\n",
"
Count
10 Tasks
10 Chunks
\n",
"
Type
float64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X, y = dask_ml.datasets.make_regression(n_samples=1_000_000,\n",
" chunks=100_000,\n",
" n_features=20)\n",
"X"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:06.026741Z",
"iopub.status.busy": "2021-01-14T10:56:06.026352Z",
"iopub.status.idle": "2021-01-14T10:56:07.699274Z",
"shell.execute_reply": "2021-01-14T10:56:07.698858Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LinearRegression()\n",
"clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n",
" subestimator,\n",
")\n",
"clf.fit(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:07.704652Z",
"iopub.status.busy": "2021-01-14T10:56:07.704234Z",
"iopub.status.idle": "2021-01-14T10:56:07.853949Z",
"shell.execute_reply": "2021-01-14T10:56:07.853577Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([-209.83798979, 153.34999104, 79.37186412, -228.01129689,\n",
" 333.11851764])"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.predict(X)[:5].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As usual with Dask-ML, scoring is done in parallel (and distributed on a cluster if you're connected to one)."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:07.856816Z",
"iopub.status.busy": "2021-01-14T10:56:07.855599Z",
"iopub.status.idle": "2021-01-14T10:56:09.500287Z",
"shell.execute_reply": "2021-01-14T10:56:09.499790Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"1.0"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.score(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The dangers of non-uniformly distributed data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, it must be re-emphasized that your data should be uniformly distributed across partitoins prior to using these ensemble methods. If it's not, then you're better off just sampling rows from each partition and fitting a single classifer to it. By \"uniform\" we don't mean \"from a uniform probabillity distribution\". Just that there shouldn't be a clear per-partition pattern to how the data is distributed.\n",
"\n",
"Let's demonstrate that with an example. We'll generate a dataset with a clear trend across partitions. This might represent some non-stationary time-series, though it can occur in other contexts as well (e.g. on data partitioned by geography, age, etc.)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:09.503571Z",
"iopub.status.busy": "2021-01-14T10:56:09.503136Z",
"iopub.status.idle": "2021-01-14T10:56:09.505682Z",
"shell.execute_reply": "2021-01-14T10:56:09.506175Z"
}
},
"outputs": [],
"source": [
"import dask.array as da\n",
"import dask.delayed\n",
"import sklearn.datasets"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:09.508051Z",
"iopub.status.busy": "2021-01-14T10:56:09.507646Z",
"iopub.status.idle": "2021-01-14T10:56:09.510426Z",
"shell.execute_reply": "2021-01-14T10:56:09.510934Z"
}
},
"outputs": [],
"source": [
"def clone_and_shift(X, y, i):\n",
" X = X.copy()\n",
" X += i + np.random.random(X.shape)\n",
" y += 25 * (i + np.random.random(y.shape))\n",
" return X, y"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:09.513210Z",
"iopub.status.busy": "2021-01-14T10:56:09.512264Z",
"iopub.status.idle": "2021-01-14T10:56:09.521381Z",
"shell.execute_reply": "2021-01-14T10:56:09.521774Z"
}
},
"outputs": [],
"source": [
"# Make a base dataset that we'll clone and shift\n",
"X, y = sklearn.datasets.make_regression(n_features=4, bias=2, random_state=0)\n",
"\n",
"# Clone and shift 10 times, gradually increasing X and y for each partition\n",
"Xs, ys = zip(*[dask.delayed(clone_and_shift, nout=2)(X, y, i) for i in range(10)])\n",
"Xs = [da.from_delayed(x, shape=X.shape, dtype=X.dtype) for x in Xs]\n",
"ys = [da.from_delayed(y_, shape=y.shape, dtype=y.dtype) for y_ in ys]\n",
"X2 = da.concatenate(Xs)\n",
"y2 = da.concatenate(ys)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's plot a sample of points, coloring by which partition the data came from."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:09.523649Z",
"iopub.status.busy": "2021-01-14T10:56:09.523220Z",
"iopub.status.idle": "2021-01-14T10:56:09.712575Z",
"shell.execute_reply": "2021-01-14T10:56:09.712919Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEWCAYAAACaBstRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAACEgElEQVR4nOyddXwcZfrAv+/MatytktQlVZo6FYoVLYWDgzvcOQ45hXOF41x+d3DHwR16uFMKpQLUjbq7xt1WZub9/TGbNMlu0iTddJN2vp9PPsmOvPPMZnee931USCmxsLCwsLBoD0qkBbCwsLCw6DlYSsPCwsLCot1YSsPCwsLCot1YSsPCwsLCot1YSsPCwsLCot1YSsPCwsLCot1YSsOixyGE+LoQYkGk5egoQohbhRDLTuP1UoUQu4QQrsDrz4QQd56u658KQohpQohdbezvK4SoEUKoYbjWGiFE7qmOc7ZgKY2zBCHEQSFEoRAiusm2O4UQn51mOTr04BRC5AghpBDC1rBNSvmylPKirpGweyCE+LkQ4qVTHOZR4L9SSk84ZOpKAv/jgQ2vpZRLpZRDmuw/KIS4oMn+w1LKGCmlHobL/wH4ZRjGOSuwlMbZhQ14KNJC9FTCMas9XQghnMAtwKkqni6l6WQggrwPnCeEyIy0ID0BS2mcXfwe+K4QIiHUTiHEFCHEWiFEZeD3lCb7PhNC/EoIsVwIUS2EWCCESGntQoEVxf7AsQcCJqVhwD+ByQHTQkXg2MuEEBuEEFVCiCNCiJ83GeqLwO+KwDmTW65WTkVuIcQbQoiCwLlfNDVTCCGeE0I8JYT4SAhRC3w7sFqzNTnmGiHExlbeg2QhxPuB+1oDDGix/6+B+60SQqwXQkwLbJ8N/BD4auCeNwW23yaE2BG4j/1CiHtae/+BiUCFlPJoi+0DAuaYSiHEe0KIpMDY84QQD7SQb7MQ4qoQ99Ww+rtbCHFcCJEvhPhOk/0ThBArhRAVgX1/F0I4muyXQoj7hRB7gD1CiIb/8abA/X5VCDFTCHE0cPyLQF/gg8D+77dcgQohsgLvdZkQYq8Q4q4m1/u5EOJ1IcQLgfdumxAir2F/YCW2HjijV69hQ0pp/ZwFP8BB4ALgbeDXgW13Ap8F/k4CyoGbMFckNwReJwf2fwbsAwYD7sDrJ1q5VjRQBQwJvM4EcgN/3wosa3H8TGAk5iRmFFAIXBXYlwNIwNbk+MYxTlVu4HYgFnACfwE2Ntn3HFAJTA3I5gK2A5c0OeYd4DutvA+vAq8H3o8RwLGm9w7cCCQH5P4OUAC4Avt+DrzUYrzLMBWPAGYAdcA5rVz7fmBei22fBWQYEZDprYZrANcBq5scOxooBRwhxm74n7wSGGckUAxcENg/DpgUuK8cYAfwcJPzJfBp4H/nbrJtYIvPxNGWn98QMtgCrz8Hngz8j8YE5Dm/yXvpAS4FVOA3wKoW9/Q34E+R/p72hB9rpXH28VPgASFEaovtlwF7pJQvSik1KeUrwE7giibH/FdKuVtKWY/5MBzTxnUMYIQQwi2lzJdSbmvtQCnlZ1LKLVJKQ0q5GfNhNKOd93NKcksp/yOlrJZSejEfLqOFEPFNzn1PSrk8IJsHeB7zYU9gln4x8L+WQgnTlHUN8FMpZa2Ucmvg3Kb3/ZKUsjQg9x8xFdeQlmM1OX6elHKfNPkcWABMa+XwBKA6xPYXpZRbpZS1wE+A6wKyvgcMEkIMChx3E/CalNLXmjzALwL3tgX4L6bCRkq5Xkq5KnBfB4F/Efz//I2UsizwPzklhBB9gHOBR6SUHinlRuCZwD00sExK+ZE0fSAvYirFplRjvmcWJ8FSGmcZgYfXh5hO0qZkAYdabDsE9GryuqDJ33VADIAQ4p8Bs0GNEOKHgQfSV4F7gfyA6WNoazIJISYKIZYIIYqFEJWB81o1fYVRblUI8YQQYp8QogpzNkuLax9pMfZLwBVCiBjM2flSKWV+CLlSMWfaTc9vJqcQ4jsBc1OlME118bRx30KIS4QQqwImmArMmXNrx5djrqBa0lIeO5ASUJqvAzcKIRRMBfBia7K0MlZWQM7BQogPA2a/KuDxEHK2fF9PhSygTErZVEme7DPgEs39KbFARRhlOmOxlMbZyc+Au2j+pToOZLc4ri+mOaNNpJT3SjOSJUZK+Xhg2ydSygsxTVM7gX83HB5iiP9hOiP7SCnjMf0eoo3jm9JpuYGvAXMwzXbxmCYPmlw76PpSymPASmAu5ky2tQdrMaABfVrIZV7A9F88gql4EqWUCZimsJD3LUzH9luYkT7pgeM/aiFrUzZjmuRa0lIeP1ASeP088HXgfKBOSrmylbFbG+t44O+nMP/ng6SUcZj+mZZydrS8dlvHHweShBBNlWR7PwMNDAM2dVCmsxJLaZyFSCn3Aq8BDzbZ/BEwWAjxNSGETQjxVWA45qqkQwgh0oUQVwozvNcL1AANoZGFQO+mjlHMWV6ZlNIjhJiA+TBvoBjT1NW/lcudityxAflKgSjMGXF7eAH4PqYt/51QBwTMIG8DPxdCRAkhhmNGMzW9toZ5fzYhxE+BuCb7C4GcwKwfwIFpvioGNCHEJbTtuF0DJAgherXYfqMQYrgQIgozzPTNgKwElIQB/JGTrzIAfhK4t1zgNszPVMO9VQE1gRXmfe0Yq5DW/8dt7pdSHgFWAL8RQriEEKOAO4CX23HdBoU8DtPPYnESLKVx9vJLTCcmAFLKUuByTIdsKeZD8XIpZUno09tECYxzHCjDtGd/I7BvMbANKBBCNIz9DeCXQohqTJ/L603kqgMeA5YHonEmNb3QKcr9AqYZ4ximg3tVO+/vHczVzTsBU1xrfBPTFFaA6VT/b5N9nwDzgd0BGTw0N9m8EfhdKoT4MmB6eRDzvSnHVKzvt3bhgC/iOQL+lya8GNhegOk0frDF/hcwlWF7QnU/B/YCi4A/SCkbEi6/G5CvGnOF+Vro05vxc+D5wP/4uhD7fwP8OLD/uyH234C5UjyO+f/5mZSyvUrgSsyAkOMnPdICIaXVhMnCoqMIIfYB90gpF0ZaltYIBDssBca21+EshLgZuFtKeW4bx+QABwC7lFILh6yRRAixGrgj4O+zOAndIbHGwqJHIYS4BtPGvjjSsrSFlLIYaDUAoSUBk9U3MENXzxqklBMjLUNPwjJPWVh0AGGWXXkKuF9KaURYnLAhhLgY019SSIgQYguLBizzlIWFhYVFu7FWGhYWFhYW7eaM92mkpKTInJycSIthYWFh0aNYv359iZSyZeWIM19p5OTksG7dukiLYWFhYdGjEEK0rLQAWOYpCwsLC4sOYCkNCwsLC4t2YykNCwsLC4t2YykNCwsLC4t2YykNCwsLizOIuro68vPzqamp6ZLxz/joKQsLC4uzAcMwWLp0KXv37kVVVXRdp0+fPsyaNQubLXyPemulYWFhYXEGsGnTJvbu3Yuu6/h8PnRd58iRI6xa1d7ize0j4koj0D1tgxDiw8DrJCHEp0KIPYHfiU2O/UGgafyuQK0cCwsLCwtg69at6LrebJuu6+zatYtwlouKuNIAHsJsPN/Ao8AiKeUgzDr9jwIEmthcD+QCs4EnA72NLSwsLM56fL7Q7dx1XccwwldbM6JKQwjRG7gMswl8A3Mw204S+H1Vk+2vSim9UsoDmM1fJpwmUS0sLCy6NRkZGSG3JyYmoqrhm19HeqXxF8xOa03VYLqUMh8g8DstsL0XzTubHaV5j+tGhBB3CyHWCSHWFRcXh11oCwsLi+7G5MmTsdvtCGG2YxdCYLPZmDZtWlivE7HoKSHE5UCRlHK9EGJme04JsS2koU5K+TTwNEBeXp5V+93CwuK0I6VsfICfDpKSkrjmmmvYvHkzxcXFJCYmMnr0aBITE09+cgeIZMjtVOBKIcSlmL2K44QQLwGFQohMKWW+ECITKAocfxTo0+T83pj9gC0sLCy6BYZhsGHDBrZs2YLP5yMpKYmpU6eSmZl5Wq4fFxfHuee22qk3LETMPCWl/IGUsreUMgfTwb1YSnkj8D5wS+CwW4D3An+/D1wvhHAKIfoBg4A1p1lsCwsLi1ZZuXIlmzZtanRKl5WVMX/+fEpKSiIsWfiItE8jFE8AFwoh9gAXBl4jpdwGvA5sBz7GbLeptzqKhYWFxWnE5/Oxc+dONE1rtl3TNL788ssISRV+ukVGuJTyM+CzwN+lwPmtHPcY8NhpE8zCwsKindTU1KAoSlCuBJgrjjOF7rjSsLCwsOhxxMTEtJoPkZSUdJql6TospWFhYWERBhwOB8OGDQuq82Sz2TjnnHMiJFX46RbmKQsLC4vuimEYCCHaFT47efJk3G43W7Zswev1kpyczJQpU0hJSTkNkp4eLKVhYWFhEYLCwkKWLVtGaWkpNpuNYcOGMWHChDazq4UQjB07lrFjx55GSU8vltKwsLCwaEFFRQXz5s1rjITSNI3t27dTV1fH+eeHjNMJK5qmsW7dOnbt2tVY4nzy5MnExMR0+bVPhuXTsLCwsGjBpk2bQlaMPXjwILW1tV1+/Y8//pht27bh9XrRNI2DBw/yzjvvtFqU8HRiKQ0LCwuLFpSWloYsJ66qKlVVVWG5htfrpba2Nug6JSUlFBUVNVNaUkr8fj+7du0Ky7VPBcs8ZWFhYdGC1NTUkIpD13Xi4+NPaez6+noWL15Mfn4+QgiioqKYOXNmY6mR1nI6NE2jOxRgtZSGhYVFj6S6upqVK1dy9OhRFEVhyJAhjB8//pRbm0opGTU0lz179jTL7lZVlQEDBhAVFXVKY3/44YdUVFQ0KqTq6mrmz5/PtddeS2xsLHFxcSHPVVU17MUHO4OlNCwsLHocXq+Xd955B6/X2/jw3b59O6WlpVx++eWdGlNKib72GNrnB3F6dWa7+7I2o5zi+krsdju5ubmnnG9RVFREdXV10ArGMAy2bdvGpEmTSE9PJz4+nvLy8mbJgqqqMnTo0FO6fjiwlIaFhUWPY9euXfj9/mYPX13XKSoqoqSkpFN5EfqX+WiL9oPffFAn1qqcczyO+tE5pE8c3CGzVF1dHYcOHUJKSU5OTuPqpKamJmS+h2EYjb4SIQSXX345y5Yt48CBA0gpSU1NZfr06bjd7g7fV7ixlIaFhUWPo7i4OGSNJ4Dy8vJOKQ3ti0ONCsMjNBYkHaVG9SMOHME4spmcnBzOO+88FKXt+KGdO3eyfPnyRuWwcuVKpkyZwrBhw0hNTQ1ZakRV1Wbl051OJ+effz6GYSClDGvnvVPFip6ysLDocSQlJbX6IO20o7rG2/jnsoQCKm0+NEXiFwa6rnPo0CG2bt3a9hA1NSxfvhxd19E0DU3T0HWdFStWUF1dTVxcHP369Wvmd1EUBafTyZAhQ4LGUxSlWykMsJSGhYVFD2To0KFBD1NFUUhMTCQ1NbVTY4ok0/TjEzoFjjpkCytSQ4JfWxw4cCDkdikl+/fvB2DmzJmMHz+euLg4oqKiGDp0KNdccw0Oh6NTcp9uLPOUhYVFj6KutADNU8cVV1zBsmXLKCoqQghB//79mTp1aodbrDZESNkuHID/ze3ohobZXTo4T8Pv97c5VoM5qSVSykazlKIojBw5kpEjR3ZIzu6CpTQsLCx6BLUl+Sz700NUHN6NUBTsrigmfeMJ0q+4ot0FBZtSVVXF559/TkFBAQAZGRlMu2IM7pUFROs2qtXmCkJRFHJyctocMzs7m7Vr1wZtb8+5PQXLPGVhYdHtkYbB4l/dSvmBbRh+L7q3Hk9lKUv/+AC1RUc6tbp47733KCgoQEqJlJKCggI+2PgZttvHct7cS7DZbI1Ob5vNhsvlYty4cW2O21qJkdzc3G6RYxEOLKVhYWHR7Sne9SWeihJki8gjQ9fYu+DVDo934MABNE1rZkqSUjbWecrIyODaa69l5MiRZGdnM378eK677rqgxD4pJRUVFVRVVSGlZMmSJUHRUYqidFipdWcs85SFhUWXIqWEOj84VIS9c5FA9eVFEOLBK3WNmuJjHR6vuro6pH/C7/c35kvExsYyceLEVsfIz89n0aJF+Hw+pJRERUXh9XqDjjMMgwMHDrQ5Vk/CUhoWFhZdhr6/DP8HuxvDWZUhKdivGIJwduzRkzxwNEaTkh4NqE43GaOmNNvm8/nwer1ER0e3mlORlJSEzWZrViYEwG63k5ycfFJ56urqmD9/frPzq6urWz3+VEubdCfOnDuxsLA4DUhgJfABYACXAtMxo42aYxTV4n9ta2PCHICxqwRfnR/nzWM6dNWYtF70m34lB5d9iO6tB0CxOXAnpNJv2pWA6adYtmwZ+/btQwiBoihMmjQpZOmN3u5kYhxuqvRaDHkiqikmJoY+ffqcVJ5du3aFjJJqMEM13Wez2cjNze3Q/XZnLKVhYWHRAZ4APgLqA69XADOBX9JScWirjoDWIvtZl8ijVRhldShJwYX/ZI0Xo6gOkehCSWxeMmP8nT8nZfBYdn/8Epqnjj4TL2LYlXdgc5njfPHFFxw4cKBZpviKFSuIjo5uVATSr+N7fSvyUCWz1Uw2uIo46K4Bp8rAgQMZP378STO+wXR4h8pIF0LgcDga9xmGQf/+/btFzahwETGlIYRwAV8AzoAcb0opfyaESAJeA3KAg8B1UsrywDk/AO4AdOBBKeUnERDdwuIsZS8wD/A02VYPLAG2ASOaHS1L60KlOoAqkBVeaKI0pCHR5u1G31wANgV0iciOx3HtCITD9IMIRaH/zLn0nzk3aEiv1xukMMBcfWzYsKFRaWiLDyAPVYAmcWgw0ZvGxJp01NEZ2KcGZ2S3RlZWFrt37w4ybymKwqWXXorf76e2tpa0tLRWq9b2VCIZPeUFZkkpRwNjgNlCiEnAo8AiKeUgYFHgNUKI4cD1QC4wG3hSCNG98ustLLop0jCoOLybyqN7Q5pV2sdKINivYH6VVwRtVfrEgxoiakgzUNKim29afQR9ayHoErw6aAbyYAX++bvbJVl9fX2rEUo1NTWNf+sb80Frcf+6RN9c2KH3JScnh/j4+GZZ6Tabjb59+5KSkkJmZiYDBw484xQGRHClIc3/UMN/0x74kcAczPUuwPPAZ8Ajge2vSim9wAEhxF5gAuYn2cLCohVKdm9g2Z+/hb+2Cgm44hI599t/I6l/R+3s0ZiPjJaKwx7Y1xzbpN7oX+aDoZ1YcdgV1FEZiJjmJTP01cea+T7MjRJjaxHy8iEIte35bWxsbEilIYQgPT39xIaW12igpRntJCiKwpw5c9iyZQt79uxBVVWGDRt2RpmhWiOieRpCCFUIsREoAj6VUq4G0qWU+QCB32mBw3sBR5qcfjSwLdS4dwsh1gkh1nWHTlcWFpHCW1PBksfuor6sEM1bj+6tp7b4OIt/dRt+T0d7Xc9qZbsALgreGuPEcdc4lOGp4LZBogt1Wl+UUWnI+hbhrt5QKxhMZdOOB7qqqiEbMNlstmYJeSInIfQd9InvcC6FzWZj7NixXHfddVxzzTUMHz68Xf6Qnk5E71BKqUspxwC9gQlCiBFtHB7qPxpyPSmlfFpKmSelzOts8TILizOBw8s/QhrBDltp6Bxd/WkHR0sA/gBEYa4sogE38DgQ+numJLpxXJOL81uTUTJj0T8/hP9/W/D+eSX+T06YypTWHuaJrnaH544YMYJZs2aRkpJCVFQU/fr1Y+7cuSQknBjbPnsQuGwnzGaqAKeK/bJB7bqGRTeJnpJSVgghPsP0VRQKITKllPlCiEzMVQiYK4umsXC9geOnV1ILi55FfUUxus8TtF33+6ivKOnEiJOAT4H1mCG3eYCrzTN0zUfh398lviIBVXFAwFmtf3kckeDENrEPtgsG4DtYYa4qdGlOEW0K9sva75wG09fQtMaTpmlUVVURHR2NqqooKVE4vzEBbf1x5PEqREYMtrxeiFhnh65zNhPJ6KlUwB9QGG7gAuC3wPvALZixfbcA7wVOeR/4nxDiT0AWMAhYc9oFt7DoQaQOy8PmikLz1DXbrtjspA1ru45S6ziBKSc9qoHV//gRoysuNBVGU/wG/hWHsE3sg5IchfO+CWirj2IcqUSkRmOb1BslNdhX0h4Mw2D16tVs37690ew0duxYxowZg4hxYJ+R06lxLSK70sgEng9EQCnA61LKD4UQK4HXhRB3AIeBawGklNuEEK8D2zE9cfdLKUO37rKwsAAgY8RkkvqPoHTv5sYVh+p0kz5sHMkln8GyB8GdDBO+CQMuCPv168qKyF/3OWMGzg65X6usYd+8V+kbMxHjQAUi0YX9ssEo6TGndN3169ezY8eOZiG4GzZswOVyMWzYsFMa+2wnktFTm4GxIbaXAue3cs5jwGNdLJqFxRmDUBRm/ujf7Fv4Ovs/ewchFPrPuJIBe59ALH4ZtECS3v5PYfqPYdoPwnr9msLDSJukzldKjDOt2T4pDSrqDpG8OgvNeQihgzxSiW97MfZrhqMO7njLVnNcydatW4NyKDRNY+PGjZbSOEW6hU/DwsKi61BtDgbPvpHBs280N6x5EioPnlAYAP46+PyXMO5uiDp57aX2EpuZje73sTH/FUb0u5PtMdUUOzzE6DaGVcdi6NUkq25Ew4JAYpqtPtyN8q3kTlWHbWi1Gor6+vqQ2y3az5kfH2ZhYdGcXe+bSqIlqgOOrgrrpdwJqfSdfAnFajkfJB9kb1QllXYfx5y1LEw6Rl1SFooIMXf1asiKYAd+e1BVlejo0L6Q9hQjtGgbS2lYWHQLtgK/Bn6IWQihC911MekgFKSEgtoo1hWksbEohcp6xfRvhJmJ9/4a+5SvIhX1RHlzIUC1sSGxBhkqct6QHa6E24AQgsmTJ4fM2Zg0aVKnxrQ4gWWesrCIOC8C/8IsxyGBpZiVdf4CdEGlnAnfRG57k+VHEjheE4MuBQLYXQ7n7DrMwD7hfbAqqo06NRqU4JWDTzGoVX3E6E1CXlWByElARNk7fc1+/frhdDpZt24dVVVVJCcnk5eXR3fJ2/J6vUgpcbnaDlfujlhKw8IiopQBTwG+JtvqgY2YymNmmK4jgXeBl6FXNfnTZpK/bS+6VBr36gZ8+dzj9Jl0Ec6YhDBd18TtduPxBCsNoapEjcuB9YFChYZEpMXguHr4KV8zKyuLK6+88pTHCSc1NTUsXryYoiIz/SwxMZHzzjuPpKSkCEvWfizzlIVFRFlL6LlbPaaZKlz8GfgjZuHoUg5tPYxmBH/9db+Xd++dyup/PoqvtipsVx89enSQuUhVVfoPGEDU7CE4H56M/Su5OO4ch/OOcxDuzq8yuiuGYfDee+9RWFiIYRgYhkFpaSnvv/9+yI5/3RVLaVhYRJQoQlfIUZAymurCGmpLQzitO0QF8CZNS5qr9lYuCxh+gwNL32fhz24K6sndWQYNGsTo0aNRVRW73Y6qqvTp04fp06cDIKIdqAOSOp3M1xM4fPhwY2vYphiGwZ49eyIkVcexzFMWZxfH18P8B+HYWnDFw4QHYfoPQYlUlf2JhJq7ScPOJ4/Hc3TTByAlCb3jufD704jP6kyp7b2Ag6YmsP4zbRxcqqH7Qp8hNUlt8REKtq4kc9TUTlyzOUIIxo0bx8iRI6msrCQ6OpqoqOAmTGcyNTU1IRs3aZrWZqvY7oa10rA4eyjZDf+dCUdWgOGHuhJY9gTM+0YEhXIAfwPiaCgCKKWDlc9N5dDaWHSfju43KD1Yzns/WIDubyOqquIQHFoKdWUtdmQAzavKpgxWGT7XgWKn1aeAofmpPLK3szcWEofDQWpq6lmnMABSUlJCVsG12WzdxkHfHqyVhsXZw7Inmie0AWh1sOkFmPUYRHcuA/nUGQl8gllKrZ7tH8exff4+moXdStC8GofWHaP/5L7NT/dWw+vXwqHPQXXCIAdcOhjcPhBpwH2Ba2yiqfIYcXUs/aZPYcvrH3NohR+jRbVyxeYgLqtfF9xvz8UwDGpqanC5XDgcjpOf0IT09HRSUlIoLi5uXHEoikJ0dDT9+vWc99lSGhZnD/nrIVS5MtUJZXsiqDTAbGRkmoEqj69D9wXLqWsGtSUh/Bvv3wUHPwPdCwNdcGUvcDT4LwqA3wAPYq5mlmIuLaKBrxKdMpO821eTv7EYb7VEBlwYQhW4E9PIGH3qpqkzhZ07d7Jq1SoMw0BKSb9+/Zg+fXqQg781hBBceumlbNy4kV27dmEYBgMGDGDcuHHNOgB2dyylYXH2kD4KirYFKw7dC4n9ATCkwebiTewu302qO5UpWVNw2k5vLH3GsDR2LNiL5mnZf1qQPqSFYvPWwM53zXsAuKAXOFo+gDzAf4CPgUPAdzG7CrwIPIvNdRUXPbaHtc+somCzhqI4yD7nYkbf8ShKxHw93YvDhw+zYsWKZuVJDhw4AMCsWa01pwrGZrORl5dHXl5e2GU8XVhKw+Ls4dxHYcfbzUto2NyQey3EpOPVvfxk+Y84WHkAr+7FqTp5ZsvTPDH9d/SJ7dv6uGEmZ2Jv4jNiqThW1ejDsDlUMnPTSR3UImPbV30iyxogsTWTSSlmceifYhaP1jGTCQHeJzr1MWY8+jTa4v3oa46Zu/61DW1GDrZJfVoZ8+xhw4YNQfWsdF3nwIEDeL1enM6zpx+H5Qi3OHtIy4WbFkD6aECAIwYmPABXPgPAO3veYn/FPjy6B4nEo3uo8dfw+7W/O61iKqrCnCcuYszVw4nLjCGhVxx5Xx/NxT+aGVzALyYDopqsPipaCYciGSjEjKRqafryAK+gfXHIVBh+w2yG5NXRlhxA25gfrlvrsdTWhm6NK4QImbTYFWiaRn5+PiUlJUFhu6cTa6VhcXbRdyrctxEMw5yhN3kILz68CJ/R/KErkRyrOUqZp4wk12nI2pUSSnZiN3Tyrh9J3tdGt328EHD5P+GN60DzwKJjcFU/cDSdD7qAe4BqzK98cCKZlBXoK4+YCqMpfgNt6SFsYzJP7b56OOnp6dTW1gY9rIUQxMScWu+P9rBr1y6WL1+OEAIpJVFRUVxyySXEx8d3+bVbYq00LM5OFKW5WecknJaZXcEm+NtAeDoPnp0Mf+4Dh5ef/LzBl8HtyyD3OigfANuHg9EQwpkKfA+YCwxoZQAHaDOhtXDe6tZWL2cP48aNw2azNVvp2Ww2JkyY0OVO7JKSEpYvX46mafj9/sYWtvPmzYvIisNaaVhYBDivzyze2vNms9WGQJAVk0Vye6q/Gjrs+xTK90HGGOgzJVgxHV0Dy34DpXug9ySY9kNI6g++OnjuPPCUnzjWVwMvzYaHDpw8sitzLHzllRYbJc3Tvu3AI8DjnCiO6AQSwXY9xO6AquBViEg/vVnaUkq8Xi82m63dkUldTUJCAnPnzmX9+vUUFBQQHR3NmDFjmvUj7yq2bdsWMinQ6/VSWFhIRkZGl8vQlO7xH7Gw6AZcPega1heu43DNYTyaB5fqwqbY+G7e909+cnUB/OdcqC0CQwOhmIrjpgXgCCSy7Z5nmpH89YBphmLb63DXasjfYJ7XEkOHLS/DpIc6cUehVlKXAtnAK0ARZq/vryBEDLYLB6C9v7O5icqmYL+gtRVK+MnPz+fzzz+npqYGgL59+zJjxoxOOZoNw+DYsWN4PB4yMzNP2YyUkJDA+eeHbCrapdTX17e6ojhd/pSmWErDwiKA0+bidzP+yIaiL9lTvpsUdypTe52L2+Y++cnv3W5mZMsmD/7j6+Gzn8NFvzN9FR/e1zxyS+rmamLho5AzE0PzBtuLtXqoPn7Sy2s+D8U71iMUhdRh41BtbSWe5WL27miOLTcN4bShfX4AWVaPSI/Bfl4/lD6nx25eWVnJ/Pnzm0UpHT58mI8//pg5c+Z0aKzy8nI+/PDDxrEMw2D48OFMmjSpU90A26Kurg6/309cXFzYxwbIzs7m+PHjQdFbhmGQnp4e9uudDEtpWHQ7pJR4qrzY3XZsQTkHXYsiFMal5zEuvQNx9P562L+wucIA0D2w8XlTadSVmquQICQcWsrR+DtI1wQtq0wYajRK9ow2L39s/RJW/O17AVOYRAiFc7/zNzJGdLwvhjowCXVgZMp0b926NcgM01AJtqysrN3lw6WUfPzxx0GtXXfs2EFmZmbYTEp1dXUsXLiQ4uJihBDY7XZmzJhB377hDc8eNGgQ27Zto7KysvH9sdlsjB49Gre7HROaMGMpDYtuxf4Vh1j+9Dq8NV5AMHhWf6belYdqD1YeFbU+Pt9ZhF8zmDI4lazE0/8FAjDTqFtxSDbU5nDEtO54j0rh87c0psTm0jt2K3bV9Kn4dQeV/mxSBs5u9dJ1ZUUs/8u30X3NzRRf/O4bXPXUZziiO1Pg8PQjq72UbT2EFMHvo6IoVFdXt1tplJaWhuwFrmkaGzZsYPfu3VRVVZGVlcXo0aNbbQ3bprxSMm/ePCoqKhpNR5qmsXDhQq6++moSEhI6PGZr2Gw25syZw86dO9m/fz9Op5Pc3Fz69IlM/oylNCy6DfnbiljylxVo3hOzzd1L9qN5NWZ9q3k5i8+2F/KztzY3hiD+bcEubp/Rn1unB9vfdc1AUQRCCb/pAABHNGSOg2NraKY8FDsMm2v+bXfByK+b/gmtyQPeHo0x+XvULK7j05IHGZL8OcOSP0MIg91l57Kz4kLuCFHkroHDKz5CyuDy5QI4snoBA2Z9JSy32NX4Xt9KWq2ToqhadKW54tB1vUO9vTVNa9VMVFxcTHFxMQAVFRXs2bOHq6++mtjY2A7JW1xcTHV1dZCvQdd1tm3bxtSp4S2/YrfbGTlyJCNHjgzruJ0hYiG3Qog+QoglQogdQohtQoiHAtuThBCfCiH2BH4nNjnnB0KIvUKIXUKIiyMlu0XX8OXrW5opDADdp7N/+SE81Seieqrr/fzsrc14NQOPX8erGfg0g/9+vp9d+ScaBxXvK+Wtb3/EM1/5H89e9wqf/W0l/voWVfnCxVX/BVcC2ANOb3sMxPWCC544ccyl/weDrwCbC5xx5u+JDyHG3YEz2oFEZWfpLN7Z/Uve3vVrthbPxp3Y9krBV1uF4Q++J0PX8NfVhPEGuw6joh5ZWMuQ2gRsiGZ6V0Whf//+HXJit7dirGEY+Hw+1q9f31GRqaurC6mYpJRUVYWveVV3JJIrDQ34jpTySyFELLBeCPEpcCuwSEr5hBDiUeBR4BEhxHDgekwvXhawUAgxWMpQFegsujvlnnJWHF+OV/eSl55H37hsqgpC9xRQbAp15fW4Ys0ImmW7i1FCrBr8usHHm48zJDOO6uJaPvjhp/gD9Zt0v8HeLw5QU1zL5b+6oFMy+3Qfx2qOEueIDw7BTR0GD+2HzS+ZUVG9xpt5E/YmJjO7G657HWoKoeooJA0CVxwCGPuVXNa9srmZ0rQ5VcZdP6pNmTJHT2XXvOfQvM3NMUJRyRg1pVP32VlKSkrYuXMnfr+ffv360bdv35ClwIOo10ARuDSVy0qy2RBbwnFnLTZDYaiSxtgZbft0WqKqKjNmzGDJkiWNxQVVVQ0Ztiql5NixYx0aH0zFZIRoUKWqKr179+7weD2JiCkNKWU+kB/4u1oIsQPoBczhRGPk54HPMIPL5wCvSim9wAEhxF5gArDy9EpucaqsPL6CP677PSAwpM7/dr7EJTmXkjNkGNVFtUij+ZJfGhCbfmKmqRuh/QcS0HVz39Z5u9C15l9q3W9QsLOY8iOVJHYwIujjA/P577ZnAdAMjeHJuTwy/lFiHE3MGu4EmPjNkw8Wk27+NGHUVcMxdMmGt7ah+3XsThvjbhjNkPPbDndNGXIOmefMIP/LzxsVh83pJvvcy0noO7hD93gqbN26lTVr1qDrOlJKDhw4QGZmJhdffPFJFYdIi26MDo4x7EyrDGSfqwJ1cp/2KZ4W9O/fn6SkJHbu3EldXR2ZmZksX7485IPe7XYjpWTz5s1s2rQJr9dLQkICU6ZMoVevXiHHj46OZsiQIezevbsxqklRFNxuN0OGDOmwvD0JEckaJo1CCJEDfAGMAA5LKROa7CuXUiYKIf4OrJJSvhTY/iwwX0r5Zojx7gbuBujbt++4Q4cOdf1NWLSLOn8dt3x8I169eRKZU3XyvX4/ZMuv9qJ5tUYThc2pMvYrIzjnuhO23NIaL3P//AW+FkrBZVf5683jGN03kXk/X8TRDcE1k2xOFWesE0+lh9j0GCbcOIZ+LftTNKH8aCUf/30xFTur0W06RwfuY2feZhSHIDc5l19NffwU3o1gDN3AV+fHEWVHUdv3sJSGwdG1CznwxXsIRaX/zLlknROiTlUX4fF4ePnll4Nm8jabjfPOO69dvSK0LYVoH+46kSNiUyDKjvOevLD1C1+wYAGHDx9upjgasrrr6urYunVrs7BWVVW5/PLLG8NapZQcPXqUHTt2oGkaAwcOxDAMtm/fjs/nIzs7m7Fjx+Jynd6qyF2FEGK9lDIojDDijnAhRAzwFvCwlLKqjQ96qB0hNZ6U8mngaYC8vLzIa0WLRjYUfYkigh+GPt3HGv8Kvva7m1n9wgYKdxbjTnAx5ppcBp/Xv9mxyTFOHp49hL9+vAvNkEilFqcdZucOZlSfBADSBqeQv7UQvUUtJc2ro3nNXImKo1Us/tNyZjxoMHBaTpBMdRX1vPv9j/HW+lBQUPwKfXcPJLoynrUXf8b20u2U1JeQ4g5fHw5FVRrNcO1FKAp9Jl5En4kXhU2OjnDk4AGk5gPRPMJN0zT279/fLqVhG5mOkuRGW30UWeVFGZiELS8L4QqPwgCYOXMmCxcuJD8/H0VR8Pv96LrOypUrQybP6brOkiVLuOKKK4iOjmb16tVs3769UbEUFBSQkZHB3LlzT5uC7g5EVGkIIeyYCuNlKeXbgc2FQohMKWW+ECITM20V4CjQNMasN2ZTAItuglFVRdXvfk/9e++DAPecq4j7/ndRmkSmyNZCUwP7krITuOQn5530WleP70v/TIM/b/g9FfoRVKFwwJXG3orvMihxMLmXDGbrhztNE1Ub0wbNp7P6+Q0hlcb2+bvRfTqiyXxF1W0kF6YSUxGHnqJR4S0Pq9Loiez56HkMVy+wBYdFn6y7XUlJCUuXLqW0tBSAtLQ0pl81Pawhq01lufTSS6mqquK9997D7/eftHZTVVUVr776KlOmTAkq56FpGgUFBRw5ciTsuRndmUhGTwngWWCHlPJPTXa9D9wS+PsW4L0m268XQjiFEP2AQZj9MS26AVLTKL5qLrUv/w+jrAyjtIzal16ieO41yCZftLFp56CHCBF1qE5m9J7Z7uvphs7ftv2cSuMwEh1N+jlee4wfL/8hFd4KohLdXP2HS+g7rheqQ8UV1/rsvaY42I8CULy3LGilAmAoBjEVcRjS6GCfDR04BlR24JzuTV1ZIdUbFpgZ7y0Q0mjTvl9SUsK7775LcXExhmFgGAYFBQW88cYbjUqkK6iqqgrKrm4LXddZsWJFyH2apnG2mb8jWeV2KnATMEsIsTHwcynwBHChEGIPcGHgNVLKbcDrwHbMFmT3W5FTp8bewmq+/8qXXPb7Jdz571Ws2FPc6bE8ixejHz0GviYVUX0+9MOH8S75rHFTtD2ah8Y+jENxYFfsKCg4VSez+sxiZErbkUJN2VD0JbX+GowWCkg3dBYdXghAfFYcl/zkPO584wZuefFaYlJDJ3G5E1whczhSBiSh2IO/Ioqh4EvycPPwW3Cq7TUlLQZmA18N/P4WZqnynk1daQGqqhK18V3we0DzguYDXSOueCdpaWmtnrtmzZqQjmkpJatWreqwLHv37uX111/nueeeY968eZSUlIQ8rqamJmzVYRVFOWN8GO0lktFTywjtpwAIWRVMSvkY8FiXCXUWsaegirueWYNX05ESSmt8/PC1jXzvsuFcNjZ0xEhb+LdtR9YF96+W9fX4t2/HdcGJf+m03tMZnjycZceW4tG95KWPZ0DCyYvi1fs0Fm0r5EhpLR73PvQQDxyf4aOgtiDk+XlfG8Wyf65pd1jr8NmD2PLBTowmJi5D1fH18vDAxQ90oNTIdsyOeU2ztldhtl39VzvH6J7E9xqAoftR6/OJWfo0enI2UrVhq8xnwPQr2jy3IckuFEVFoUqutM6WLVtYu3Zt4wri2LFjvP/++8yZMycoMTAtLS2k0mjwS4TapyhKq+ec6dFSLbH6aZylPPnpHrx+vZlVweM3+NuCXa2GtLaFrW9fRFRU0HbhdqP2DS53kOxOYc7AuXx1yPXtUhjHyuq4+i9L+eO8HTy/9ADzVun4tOCFpkt1kZucG3KMIbMGMOXOvMaVhSvWyYSbz2H47EEhj49OiuKq315M5vA0hADVoTLiwqHc/4c7O1abihcJbnzkB7Ziuup6LvaoGIZefjuq042QOraS/TiK9uJQYNicO0KeI3UD41gV0a7gz0sDHZm967rOunXrgkxOmqaxdu3aoOOTkpLo27dvs7LriqIQFxfH+eefH7Icu6IoXHrppbhcLux2O3a7HZvNxsyZM4mL6xmlWsJFxKOnLCLD9uOVIf3D9T6dilofyR2M4HFfegmVv/wVsr7e7IoHoCiI6Cjcs1uvndRefv3uVirqfI1KrqYqBVt1b1zxxzAwM6Jtip1kdzJTs85tdZxhFw1i6IUD0X06qkM9adRLUt8Ernz8IqSUpxAhc5zQ3ng7UIwZ09FzGXndA8RmZrPj/WfxVJWRnjuRUV99iOiUrKBjtW1FZmithFG2KD6PLUe2eFuFEIwdO7bd1w/VUa+B1kxU559/Ptu3b2f79u3ouk7//v0ZO3Ysdrudo0ePsmfPHqSUjTkiF110ERkZGdx4440UFhai6zoZGRndpt/H6eTsu2MLAFJjnVTWBZefEECMq+MfC+F2k/r+u5R/6zv4AmUZHOPzSPzznxCnaPP1+nU2H6kI8rVW7L+MhKzNZPffi9/QmNZrGtcO/ip2te0wTSEENmfH7vHUQirHAbuBlu+3j9a66UnDQB6vAUUgMmPCGNJ5ENM05gbOAzo2S5Y+M3tbNImUEkLQb/oc+k1vu3y5UVSD9v5ODioVfBlXSo3qxyEVfBiNhuoGhdERk09Dcl4oWqsppSgKI0aMYMSIEUH7pk+fzogRIzh27BhOp5OcnJzGKDBFUcjMPLtb31pK4yzl9hkD+OU7W/A0iQ5y2hWuGNsLZ4iKsu3Blp1N6ttvYtTWAqB0onpoKHyGD8VRie6NAtn0I6tilI/nXxc+2vrJNUWw4y2zfPngyyAlEvbnr2EGAdZgRlCB2bf7a4R6aOv7y/C/tR0C2e04VRxfHYmS1bGies2RwF+BNwJ/q8AfgN8DJy+hbhRU439/F7Io8L8dmIT9yiGIqLZDapuirz3OQbWC5fGF6IFqtj5hoErBmIG59B05iHhHFGJbKf4Pd6H0iUfNTUOc5PNot9uDsrPBTNwbN25cu+VrSlJSUrur6p5tWErjLGVWbgalNV7+uWgvumEgJVw6OouHZw895bFPpixq/bXMP/ARXxauI9mdypUD5jAoMdivYEiDl3a8yAf73iNpqIFhSGqLxlJXMAkQ2FXBxSODTSCNbH8b3r7RLElu6LD4xzDxIbjwN6d4hx0lBfgfZr7pKiAeM3Aw2Gwnq734X9vavHueT8f34kac355y0gdo66zHTIlq6Vv5PrAAU4mFRtb48D2/EZoEEBh7y/C9sAnHPXntXgXJai8bYkobFUYDupDsOLyXscNG4vvPl6ay1AyMrUXoXxzCcec5J1VOU6ZMQVVVduzYgWEYOFQ7k4aOoVdWx4M6LNrGUhpnMddOzGZuXh+KqrwkRttxO7r+41Dtq+bhJQ9Q6a3EZ/gQCFblr+D+MQ8ws0/zpL539rzFB/veM0uOCDPhODp9A8JwYVTk0TsxinvPD+3ExlMJ79xkdr5rypq/wZAroO/pLeYH6cBPTnqUtqUQQgUiSDB2laCO6Gyntg+A4B4Tpl1oLTCtdZk2HIcWJVswJLLCQ/2+IpTe8e1yXCuDkqhZF7rKcJ3Pg++9Hc0UE34DWeVFW3IQ+2Vt19FSFIXJkydzjtKL+sX7cCoqorAK78qVOG4chZJ2aq1eLU5gKY2zHJuqnNbmRe/ufYcKbwX+QHMiicSre/nnpieZ2utc7MoJf8Tbe98OqlElFI2k3hv4zsV3MqF/cshqtwDs/TiorAVgmqk2vxQBpdFOav0nzFJN0Q1k7amUdW8rma3tdCdZXBckU5XqY2lcAeVLdoIQJCUlMWvWLGLLJfrKI2YpkAFJ2Cb3QcSYqwR1VAbR6xzU4Au6RpQ7Cg6HUGqGRN9ZfFKlAWAcq0IuPojLL4CAkvPp+F7ajPPhyV3XT+Uswwq57WZIKZHeliaEM4c1BasaFUZTJJLDVWZm7foDpXz9yeVUeUP3JfDLOiYNTEH3auxYsIflT69l58K9jWXQzQGDczhO7Ou+OaFKvwRwhPhaKgIlJ+EURr4E0/ndEh0Y37ZMveOgSZKjhsHHSUcoVT0YUmIYBiUlJbz31jvUvboJY28ZsqgWfc1RvP9ai6wxP8/CrjJhxhTUFrXHbDYb48aMbb3cSzsKN9bV1VG2eh8yRBg2Ph3jUMVJx7BoH9ZKo5sgdZ2qP/2Z2meeRdbVofbuTfwvf4H7ws71fjhVjpfX8dLyg+w4VsmAtBi+fm4/+qWe+hI/1hE6Wkc3dKLtMWw9UsF3Xv4Sj98gKTEJm7ss6Ni+cX2pLqzhne99jN+joXk1bC4ba1/exNV/uITo5CgYOPtEq9Wm2KNg5NdO+T7CSfmhXWx5/f8oP7CNmIxsJqTejr1COeHXsCsoQ1NR0k/l/Z8KzMDsNODF/OormEmHbfug1NEZaMsOm5neEg67atCEEZSaq2sah2yVDPQHys7rEuo1tOVHsF88EICBQweDTWHNmjXU1NQQHR1NXl4eQ4YMwbthI/JQRXPlYVNQz2k9Wqm2tpaFCxdSUlKC0CX2FMGUygx6+ZrckwC87S8bYtE2ltLoJlT++jFqX3wJAr2N9cOHKb/vPpSXXsQ56eTRLeFkX2E1dz2zGq9moBuSXflVLNxWyF9uGseY7MSTD9AGVw6Yw57y3c3MTgoKOXE5ZERn8Pib6xojuqqPTSeh/4cgtMb22k7VyR0j72bpk2vwVHsba0ZpHg3dp7P83+u46NHp4E6EK/4NH9xtriwMzeyUN/oWyJ5+SvcQTkr3bGTRL25GD3Tfqyst4GPHFmbO+S1x1UmgKtjGZqKMaL0cR/sQwK+ATcAyTEUxGzh5+Khw2nDeNQ7/wv0Ye0qpdcuglqwAGpIatYWiNiSezQdhWgb2KFPpDRw4kIEDBwblvjjmDsP33AbTDBcIoRV94rFNDV3fq6FPd2Vl5Yk+3Sp8lnCcK0qzidMDznNdovRNOOl9WrSPbtFPoyvJy8uT69ati7QYbWLU1ZE/cjR4PEH7HFOnkPr6a6dVngeeX8va/cEz/P5pMfzv/lPvffzKjpd5c88b2BU7utTJjM7k51N+RZIrict+v4TSmhM2b1t0PjEZq7C5y8hNHcBto25mSOJQ/n31/0IWGVTsCne92WQlUXEItr0OvlrTAZ7VuRDMLkH3s+j+cygqD54Fu5PSmfPkkm5ZcvvI4cN8+ukCNL25CdBmCKZXZtLb23xFVFy3m9WFz3D+z14gMaft6DwpJcb+cmSlByUjts0w48LCQubNmxeUCS4kDK1LYHx1GtgV1Bn9sE8JrkoQKWpraxsbNnVnum0/DQswiooQihLSpKvt23/a5dl8uCLk9gPFNfg0A4ft1FxhNwz7OpcPuJK9FXtIcCaSE5fT+HDMSY2htOaEwtJqM6nYNxeXXeEXV83CZVfNGaoiQiuNls7OhGyY+r1TkrfL2PoqZZU+QrkWPZWlaPW1jbPz7kLl0b2s/+3dyP6zIDoJFPMRoqoqscJJltZcXs3wsrv4E/x11Sz/y7e57M/z2lSEQgjUAe3Lj6itrQ3dp1tATZxAyUnHNi4LpW/HujR2FSUlJSxatKixYGJycjLnn39+jytDYjnCuwFqRkboHUJgzx1+eoWh9Yxwh6pga/FQlrqOXlaO7ECpaYBYRyxj086hX3y/Zl/8u2cNxNmisqzLrvDVSdm4AjkKQgj6Te6L0kJ5KTaFASH6YnRbtr2BSw39vimKQHV2r+qphqGz+Fe3U19SgHvda9gPrUd4qhDeGgb2yWTOV69G7R2PLv349Xo0w8u2wvcoqtkOQF3JcepKg7spdpa0tLSQVXJtNht9Jg7DMXdYt1EYHo+HDz74gMrKSnRdbwweeP/990PeQ3fGUhrdAOFyEXP/NxAtlqvC5SLuu98J67W869ZTfP0N5I/Lo+SGr+Fdtz7omGsnZuNq8eB22hQuH9urcSYvpaT66X+TP3I0BePyyB8xiup/PHnKJadH903kd9ePJSfFdGTGu+3cPmMA98xqno9x7t3jicuIwe6yodoV7C4bCb3jmHx7NzI/nQxXAsOTS1FF84eGKgwGTZqBonYvQ0DR9rWBrocSYWi49q8gZvmzxK54lqhDa3DEReG8ZSwrK59h2cG/8tHO77G/bEnj+QEvRdjkiYmJYdCgQUGFB10uF4MHn77+6O1h9+7dQcpBSonf7+fw4cMRkqpzdK9P5VlM7EMPoiQlUfP3f6CXlGDPzSX+pz/BMar9PSYApNeLd+kypNeLc+oUlCYd0DxLl1F6221Qb/pOvAWFeNeuJfm//8U1zSzyt6egmiMlNUQ5bfg1Py6Hgl+XTBmcyoMXnyjBUfu/V6j+3e/NAoWA9Pmo/vNfEE4nMXeGrm7aXiYOTOHVB85ts0igK87Jdf93BUc35VN5rIqEPvH0GpnRs2Lx8+6h//a38OglbCtJQSCRCPql+Bh9zx+aHPgJ8BRQCPQFHkLKyZQf2E59eRFJA0bgTkg96eU8VWXs+ugFCreuJiopHaEo5G9ahqFrZI2dzjk3P0pUciurXsBfGzoEWhoGnqoTTZOypl/A1reeRJdNneKC2PQ+RKeEt27TtGnTSE9PZ+vWrfj9fnJychgzZsxJOwaebqqrq4N6qAMYhkFtoOxOT8FyhJ9BeNesofSW2xqrzEq/n/hf/JyYm24EoHDWBWi7dgWdZxs6lPRFn7JsVxE/emMTfs3AkGBXBS67yv/dksfQrObL/Py8CRj5waYGJTmZzM0bw39zZyrLfw9LfoqOnTq/istlx37rAsgcEzjgXcwaUSeCJOrL7Sx5LJraojKEoqJrPgZdeANjb/4+QnyBWV+qHrgYuApwUl9exPzvz8VfX4PhD06uQ1FwxSVx+V8/xu4KHYJbX1HC+988P+h8m9NN3h0/pd+MqwDQ/T4+e/xOyvZvQ/d5UR1OFJuDC37xIrH2DIwjlYhYJ8rAJEQ7cjDOBPbu3csXX3wR5LS32WxcfvnlbTarihSddoQLIfpJKQ+cbJtFZDHq6ym96RZkTU2z7VU//wXO8XnYhw5F27075Lna7t0YhuTx97bhbVLzyK9LDKnz/pfHgpSG0UqTHKO0FGkYCKU7Pgzewaz/VAL0Ah4EZkVUIqZ+D8bcinroC2KdcZBzHjSapSTwJM2bN8GyP1dRdayiWf7i3kWvk9T/EDnTtnKiXMhuYB7wH7a+/RS+2iqk3orvyTDw19dycOkHDLrw+pCHuBNSyJ17D9vfewbda15DdbiI7dWfvlMubTxOtTuY9dPnKd6xlpI9m4hKyqBX3vnID/fj2/OleZAiwK7guHUsSnLrfTU6Q2FhIfv370dVVQYOHNgtCg/269ePL7/8kqqqqkYzlaqqZGRkdEuF0RbtMU+9BZzTYtubmPWeLboJ3sVLQm6Xfj+1r79Bwk9/gpKYiFEWHEqrJCaSX1FPnS/4gaIbkuW7i9leup2Xd7zA4eoj9InpwyXTBtL/s+BVi5qd3U0VxpvAXzjxAD6KmdhmAyKctxGdCsOvCbHDC1Q021JXZlC23wgoDEGDp0D31rNr/lJypjV1nnuAA8Ai8jcsbV1hBNC99ZTt22o2WW6FEdd8g5RBY9iz4BV8ddX0nTyb/jPmotqbm4OEEKQNn0Da8AkAaOuOoe8ta17Dyqfjf30rzvsmtClXR1i+fDm7du1C0zSEEGzZsoW8vDxGjx4dtmt0BlVVueqqq9iwYQP79u1DURSGDh3KyJEjIypXZ2hVaQghhgK5QLwQ4uomu+JoqySmRUSQdXUnmh81RdeR1WYv6phv3Ef1H//U6IcAsw9GzDfuQ3faWu3Y5447yk9X/BVfICGv0lvB7mujubs8idxNJ5SQcLmI+8mPw3hX4UIC/6TljN18/Q/CpTQ0n86+ZQcpPbCRPuM2kzowFlfMRcBIOucAdgIxwAlfglYPqdGDGJF8PbHOTDTDw/6yz9hR9CH+2pBttYDlOGITqC0+1ubVVIeL+D6tFIBsQsaoKWSM6ljtLn398eaVewPIcg9GeT1KGOqfFRUVNSoMMB3NDV39BgwYQExMZMOXHQ4HEydOZOLEiRGV41Rpa0o4BLgcSACuaPJzDnBXl0tm0SGc085FhnC0iagTnfNi7rmb6DvvQLhdiKgohNtN9F13EnPP3SREOxibk4RNbf5wc9lVnJmfNSqMBnxovH3vaOxjxiBiY7GPGkXSs/8m6pJT79IXfrw0ffA2JzztVv31ft7+zkcU7/s3E278HVm583FGvYqh3wc8TuuFldpCAHfSdI4WJXozIfN+4lxZCCGwq24GJM9idNYN9Bofai5nA5IZevltqM42HsxCoNod9A/4JcJOqCKMYN5ia/s6yIEDB4J8Bg30tAil7kyrSkNK+Z6U8jbgcinlbU1+HpRSrjiNMlq0AzUjg7iHHzLDdgMRRyIqCufUqTjPm2m+VhRi7rgdx/TpSK8X6fWi7dyFUVAAwK++MoqhmXG47ArRThsOm8LcvN5U+gtCXvO4UUrqh++TtXM7afPn4Zo58zTcaWdw0nqHuvC0Wt0ybxe+uiIm3bIEm1NHtRsIBRTVi5TzgQ3tGEUCr2AWF5wM3Iq52L8fsweHwFh+GarS3BRkU5z0TZjIsIvSCV7RqMBcsqdcypBLb0axO7G7Y1AdThyxCaCoCKGQOnQcF/76VRwxXZPXoIxIA1uI1ZbLjkgOT2a0qoZu3yuEaGzbanHqtMenUSqEWASkSylHCCFGAVdKKX/dxbJZdJDYBx/AMWUKda+9hqytw33l5bguuqjRxyD9foqvvAr9+HEIrEo8ixZRdMUWMpYvJT7KyTN3TeJAUQ1FVR4GZcSSFONk/fw4KrwVQdeLc8R1yzIXwQjgHszOdU1NVC7MB/Kps2/pQXqN3IehKwSXGvcCCwl2DZ6g/OBONrz4AKV7juKIFQy9wsbgi7cgxP3AM4Hz/ciiDYgQfTFUpxOH8UfgEaCcE/PBnwF9EQJGX/8wQy+/lYpDu3EnphKX1Q/D0MEwUGxtt8g9VWyT+mDsLEGW1plmKlWAInBcPSxsn6GBAweyadOmoNBWKSU5OTlhuUZPQdM0fD4fbrc77N/R9iiNfwPfA/4FIKXcLIT4H3DKSkMI8R9ME1iRlHJEYFsS8BqQg9nQ+DopZXlg3w+AOzC/lQ9KKT85VRnONJx543DmhY5R8CxciFFaCk2X8LqOrKqifv58oq66CoB+aTH0a9K05iuDr+PF7c83KzLoVJ1cPegrXXIPXcO1mB/3ptFTDxAuf4bdbTcVRkhLi+DEV20Z8CqmuWwWcC1Vx4tY+LOvoXlMZaB5JZv+56euWDL2JgXzK/hHwIFIjzEfvC1XFAaIhEGYbWV3YSrHXKC5MnDGJJCee8LxrCgqKJ3tBth+hEPFccc5GLtKMA5WQLwL2+h0RIwzbNdISEhgwoQJrFmzpvFBKaXkvPPOa1eTqPbg8XjYvHkzBw4cwOl0MnbsWLKzs8MydjjQNI2lS5eyf79ZfsjpdHLuueeGVWm2R2lESSnXtNBW4aoz/Bzwd+CFJtseBRZJKZ8QQjwaeP2IEGI4cD3mNyELWCiEGCxlN26O0M3Q9u1HhiiKKGtr8e/Z2+p5V/S/kjp/LW/veSuQfia4auBc5g68utVzIoUhDbaUbOZQ1SF6xfRiTNpY1MZmTHMDP2Fi/2L45FtQtI3L4xL58vDFiBDVX4WwA5diPvxf4ERI7D7gA7a93Q/d19xnpHthzycaudcYOKICodKaD1vpL/BxO81iUVRQx2UhGsu/nHrL3q5AqArq8DTU4V0XYjpy5Ej69+/P4cOHURSF7OzssCqM119/HU+T79Ann3zCgAEDOP/888NyjVNl8eLFHDlypHG1VVdXx+LFi8OaC9IepVEihBhAYA4lhPgKEJYCMlLKL4QQOS02zwFmBv5+HrMBwCOB7a9KKb3AASHEXmACsDIcspzpHCurY3vyQGLS+9H3eHMFIaKjsbdRdkEIwfVDv8Y1g66lwltOgjMRu9q15ozOUOuv5YdLHyG/rgDd0LApNpJcyfx22u+Ic4bZVn9kJbxyBfjrALD5ShiX9BYH/juTnNvLkYZAKBLVoSDEHUAG8F9o1rXOCxRStr8IGSLyTbFBTYFBUv9+5oZ1/0IpmYfD2IJffBPJYKAaVfkY2wVPhff+ToI0DAq3rqJ072bcyRn0nXgRNlfr+RbHjx9n/fr1VFZWkpycTF5eHqmpJ89i7wzR0dEMGzYs7ONu3ry5mcJoYN++feTm5pLRWg2500RdXV0zhdGApmls3LiRiy66KCzXaY/SuB9zTT9UCHEMM/D7xrBcPTTpUsp8ACllvhCiQT32AlY1Oe5oYFsQQoi7gbsB+vYNXYv/bEHTDX7+1ha+2FWEXVXRLv42/UoO86OP/0KU3wM2G0piIu52RD3ZVTupUd03Eem/W5/lSPURNGkuhP2Gn4LafJ7c9A8enfDD4BOqjsOBxeCMhQEXg70DM9IlP21UGA2o0sOA0i84sOpzEvrtIKG3EyGmYfasWIppKmqZje0hNiuKqmMnci4a0DWITnFhRlABm54Hfx0KO3HKb544UMZA8TchY1TjmOZcqwgz3HcM4az5pPu8LP717VQc2onm9WBzutjwwhNc8PMXQ4bsHjx4kMWLFzdGNtXV1ZGfn89ll11GerrZ89w4Xo224jCyrB6lTzy2KX0Q8d0rsr/B5BOKjRs3Mnt2ZCMHa2pqUBQlZLmSysrKsF3npCEFUsr9UsoLgFRgqJTyXCnlwbBJ0H5CfepDWpCllE9LKfOklHldNZvpKby47ABLdxfh0wxqvTpe1cG+tH78a/ot4HTivuxSUj98H9HNavV0hi+Oft6oMBrQpc7q/FUYLdu/fv5r+NsAmHcfvH0T/DETjq5u/8WKtoXcLISg/6gUkvrehKJcx4kmR4k09q1uhkLu3LGojua2fdUBfSfF4Yz7ExBQBqF6noPZsKjRL3EQ0034OGYOyoPAvQQrq86z88P/Un5gG5qnDqSB5qnDV1PF8r8GF9eUUrJixYqgUFhN01i1ypwD6rtL8T2/AWN7MbKgBn39cbz/XIdRVhc0XiRpq56V338q/dvDQ0JCQsiKuUKIRuUcDk6qNIQQ3xZCfBsz/OSuwOs7hBBjwiZFcwqFEJmBa2diTpfAXFk07aTSGzjeRTKcMby19kiz0iAAfqGyauAE0nbvJunJf6BGWrHqftj6GrxzKyz4PpTu6dQwRsiHcqDvetP5xZGPQX0G7uoHN2RCjgKeCnj5MlOWk15Ih/jWVrASYkJ9QXOBFIK/cg6SB9zHtO/+nZj0PghFRXW4GHD+15lw71LM0NsA59xhtqttSVQKpDaU0H8UqATqMONF6oGtwP9Ofl/tZP/n7wb5YEBSXXCYutLm4dm6rrdakK+0tNSs9DpvV/PEP0OCT0Nb3L0qFY0dO7bVfYMGnTwpsqtxOByMGjWqWdVfMOtbjRkzJmzXaY95Ki/w80Hg9WXAWuBeIcQbUsrfhU0ak/eBW4AnAr/fa7L9f0KIP2E6wgcBa8J87TOOel/oOAEpwacb2E+xodIpo3nhvzPMmbu/BhQ7rPk7XP1iK6U1WmdCxkRWHF+B0SQ2QkFhVNroJs7wakj+OWQmQcO994qGZQWwug4Ofg4D2ujLvuMd+OAe8LaSLJg+Cuyh8g4E5sz/Ycy5joq5UH4UGErmaLjibwvQPHUoDqcZ1dSSsbfDnnmwf6Gp3GxO0/Fx/TuB3Jwi4DDBC3Av5tf31tbvq0OETsYTEFQaX1VVVFUNmXTncrmg1gd1IRS1xIyy6kb069ePvn37BiUKJiQkMHDgwAhJ1Zy8vDzi4uLYtGkT9fX1ZGVlMX78+LA2emqP0kgGzpFS1gAIIX6GWchnOrAe6LTSEEK8gun0ThFCHMUMKn8CeF2Y3sPDmLGSSCm3CSFeB7ZjRm/db0VOnZyJA5L5bEchLSuE5KREE+3sBpXxN/wHirac8A8YfvPn3dth8OXmg7Gd3DnybnaUbqfWX4tH9+BSXThVJ/ePfqDJUW+CQz+hMAAcKkzLhE0HgvwUzcjfAG/f2PYxx7+E2hKITgmxMwt4HdgP1GAWXWh+f205k1FtcP27cGwNHFpqrmiGXQ2Ohqq0Bq37Lk79qyKlROZXM/ycr7Np0ZN4vc3t5DEhSp8LIRgxYgRbt25tpjhsNps5c3e0/hkU7m7w+WzB7NmzOXToUGM+yKBBgxg6dGjQ7D5SCCEYMmQIQ4YMOfnBnaQ9d9qX5gZRP5AtpawXQrRco3YIKeUNrewKGb8mpXwMeOxUrnm28c2LhrD+QBn1fh2fZmBTBXZF4Qdzcjs9ppSSjzfn89rKQ9R4/UwbksbN0/qTGN0Jv8iW/4V+CAvg2FrIPrfdQyW5kvjnhf9m2bGl7K/cT3ZsNtN6T8dtazrzXwG2EDNl3YA0G+TMaP0CK/8EWnD0TDPsbijcBP3bCsHs3/YYbSEE9J5o/gSRjhmldajFdiemgaDzGOX1+F/ejKz20kvpR8aAX7OzdB67Cz/B5nSh2OxMeeiPIc/Ny8tD13W2b9/emD8xduxYhg4damZrD0vF2FHcvJyIXUGd3DV9vWWND1lej0hyIzrxmc3Ozu5WuRmnm/Yojf8Bq4QQDWaiK4BXhBDRmLN+i25MVqKbVx84l7fXHmHLkXL6p8XwlQnZZJ1Cgbi/fLyT99Yfw+M3Z69vrjnMom0F/O/+qcS4OhiK62iliJw0WjHztI1TdXJ+3wtCzzoASDObSIsWikNRYOz3wNVGaG7FQWjpUG+J7oPYkEF9pwGB6QC/B3Mx7gGigGzgpk6PKqXE/7/NyPL6RsuUKmwMT59DQt5oRJ84+ky+uNU+HIqiMHnyZPLy8qivrycqKqrZzNx++RD8Xg3jQIWZKa5L1PG9UMeGt2GT1A387+8yFZQqQDNQRqVjv2xIz2reFWHaVBrCnBY8B3wEnIv5qbxXStnQ1ejrXSqdRVhIjHZwx8wBYRmrpNrLO2uP4tOb992orPPzzroj3HRuB2fRefeaphZ/C2epOwkyWy+70XluAPE5TcuJSAP8nixK5a1kGLL1B0i/WebqR29lga3YIWM0pHYwua6uFAytFQd6RxkCfIjZ7a8QM/JqMqYPpXPIwhpklTfIlSF06OPKw3HeiHaNY7fbsduDJxXCoeK4YRSy0oOs8iJSohDu8OcBaYv3mwpDMxrTk42tRWjxLuzTc8J+vTOVNpWGlFIKId6VUo7D9F9YtJO2WpV2Z6QhKT1YjhCCpOyEoAfozuOV2G2Clv51r2awbn9Zx5XGkCth3F2w7p8gbKb5xeaCr33YWHgxvIwAfgj8FinB0H2U7k9m0Z+uor58Me4EF1f8+kJiUkPMmic8YMpZX276XRoQNjPkNWcGXNOBKKWKQ/DW1+D4OkBA0kC4+qUmXfs6SwzQsSCCtpD1mtk0KdS+2vCF8op4V5flZkgp0dcdb97PA8BvoK85ZimNDtAe89QqIcR4KeXaLpfmDMCzdBmVP/0Z2u7diIQEYu65m9hv3t9NGxM1p2BHEQue+ALNoyEBZ7Sdi34wg7RBJ5y6KbGukG07VEWQmdAJk5cQMPvPMPEhOPQFRCXDgIugSzPOLwUu5OCaz1jzwj4qjjaYyDS0oloW/PYLrv7DJcGnRafAvRvhi8fMKCZ3Mkx6GHpNBHeC2Uypveh++M+5UH38hMmreBs8NwMe2m++D90EpVds6PLlNgVlWA/KgwrRzwMAT7iqIp0dtEdpnAfcI4Q4BNTSGFknR7V92tmHb/2XlN12e2OTI1lRQc3f/g9ZVUX8j38UYenaxlPlZd7PF6M1+QJpHo0Pf7KIG/9zNY4o8yE+JDOWXkluDhTXNmvaZFMF1006BedgYo75c9qw8+WrviYKw0QakrKD5dSU1BKTEmK1EZsJl/0ds2TaKbBnPngqg30kuh82vwSTHkJKydE1n7JnwStonjr6TrmEgRd8FVtbfTG6AOGwYbtoANqn+048eG0KIsGF7Zzw+h26CiGEWeyxsCZ4X6/whaOeDbRHaYSYclmEoupPzbviAcj6emr++xyx3/4WSlR4eyGHk71LDyJDdO6TUrJ/xWGGXmD6RIQQ/PWmPH70xia2H61EUQRuh8qP5uTSP60Vp3YLakvrOLoxH9Whkt2nGPvRxeCIhdxrOzZbP0X8rcwwdc2g7GB5aKURLioPNzdxNaDVQ9k+ADa88Fv2Lnq9sR93xeHdHPjifS769atB7VW7GlteL5T0GLQ1R5G1ftQhKahjMxGOrq+QGy7slw7C99Im00QlCRQfVrDP7h45Fj2FkyoNKeUhgEANqO5VDKaboe0OncksVBWjoBClofBcN6S+0oMeIhFQ9+t4KpuHmSbHOvnn7RMoqfZS59XolRSF2s7ok03vbGPty5sQqmBqxjMQvwyp6gjVAZ9+D657EwYFz1OklBTVFaEIhdSoE4qlYGcx61/dTPmRSpJzEsm7fhSpg9pn2uk3uS+b3tuO1FooSwlL/rqSrz8zF1tX5bJk5YUuC+KIgT5TqC05zp5PX8Hwn/AZ6D4PNfkHObxyPv2mz+kaudpA6ROPo8/JCz9KKSnZvZGi7WtxxSfRZ9LFOKJiT4OEbaP0icdx5zi05YeRRbWIjBhsU/uiJHffyVx35KTfCCHElZjF/LMwU06zgR2YdREsmmAbOsRscNQCqesomZGtgHkyskaks/m9Hc3MUwCqTSVzROgihSmxTohtf/Jd8b5S1r2yGd1v0Me9kQFxy7ErPnPWpwVWaG98Fb5X2Czcdl/FXp56558krE/HVevGk1PLDXd8BWdJFJ889hlaQNnVltZxfEsBs386k8zcdBTRth9pzNW57Px0L56q4Ggo3a9zcPURBk7vIkXfeyL0ngRHVpy4d9UJcb1h2NUUr1qAotqaKQ0AzVvP8Q1LI6I02oOhayz944MUbV2F7veh2B18+cITnPejZ0kZNDrS4qGkRuO4KvwVcM8m2uOd/RUwCdgtpeyHmXi3vEul6qHEfftbZrvVJgi3m5g770Bxn147dEfJGplOxtBUbM4Ts1+bU6X3mAzSBofKbu44uxftRw/YxAcnLsOuhghdFQIOLGl8WeOr5u/PPEW/D0aQciSD2LIEkjZm8NF3P+OLp1Y1KgwAJGhenZf/9AbXvH8Vj6/+NeWe8lblccY4GHJB6FBkzadTU9KFBfOEgK/Pg2k/hIQcM7djwjfhjpVgc+CMSwwZPSZUG+7E8Pw/uoIDn79L4dZVaN56pKGje+vR6mtZ9scHQ5Z/t+h5tEdp+KWUpYAihFCklEsway1btMAxdizJLz6PPXc4qCpKcjKx3/0OcY8+EmnRTooQgtk/OY9Jt48jbXAyaUNSmHr3eC58ZHrYQof9Pu2E36Rlcl0rfHbwMwauGoFNtyECJTJUQ8XmsVFdELoQXmxZArrUWVuwhke++C660XoJjczcNOyu4AW3alfDpixbxeaEGT+Ghw/Ad47CxX8wo7CA9NyJ2JxRtCwLoqg2Bl7w1a6V6xTYv+StRh9MU/z1NVQc3hUBiSzCTXsMthVCiBjgC+BlIUQRZikRixA4J08mbUHP7EKr2hRyZw8md3brDZkaOFx1mKK6QnLi+5Hibt/Dtf+UbPYtPYTm0dhTdi7ZcZuCVxvSgH7nNb4sPlwasjyeItVAHF/wPp/LHFOXOpXeStYXrmNCZqiyG9BnbBbxveIoP1zRuApSHSqpA5LIzI1c7xBFtXH+z57n8yfuxVNRDIqKEIKJ9z5GXFb39Y3JNuYCLYsZWvRM2qM0NmHWWf4WZgZ4PGb2kMVZSK2/ll+t+gX7KvaiChW/4WdG75ncP/aBJpVkQ9NnbCbZeb04tO4Yh6vGsL9iAv0TV2NTNITNAUKBa15p5s8YmDWQLca+kOM5k+3o1Qaa98RKQrNp7Bt5orqNz/BxrOZYqzIpqsKVj1/Epne2sXvJAYQiGHL+AEZfNSziyZlxWf24/K8fU3lkD5qnjsT+w1Ft7Yma0oANmBHy5wCnL6S0/8y5VBzeFbTasLmiSczunm1oLTpGu/I0pJQGZgnN5wGEEJu7VCqLU0ZqGrUvvkTtSy8hvT6i5lxJzH33osScmr7/vw1/ZXfZrmbNjpYe+4I+sdlc0X8ONrV1i6cQgvO/ey7HNhWwf8Vhytx/Jn1kJQm1K8AZByO+GlRKY8qwyazN2oTrWAyqcUIpGXaDmfdMIX9rIdvm78EQBpru58DwXRwcvrvxOLviIDuu7fwRu8tG3g2jybsh8o7algghSOh78pXfCXYCD3CixqgGPARcF27RQtL/vKs5unYhxTvWo/nqUR0uhBCc++2/9ogEV4uTI1pbMgoh7gO+AQwAmjaVjgWWSym7suVr2MjLy5Pr1q07+YE9GF3qGIbRrG936d334Fm8BBryRpxObDnZpH08v9Nd+ryahxs++iqaEZzfoHvjKN9xK6P6JvKDK3PJDmOOQ1V5Fa//8n38h3SkKlGljQlfH83YuSMBM9+isriKH2z6HiV6MXqgYr5N2MiK6cXfZv39pJFUZwYaMBuoaLHdCfwbGN7yhC5BSknxjnUUbV+DMz6Z7MmX4Ig5eaiurvk4umYhxTvXE5PWm37TrzIDAiwighBivZQyL2h7G0ojHrNH5W8wO8U0UC2lLOsSKbuAM1lpeDUPT2/5F58dWYJu6GTH53D/mG/Sr8Cg6PIrwdM8v0JERZHwuyeImju3U9er8lZy6yc3h1QahuaiZMvdCAGxLjtvPjSNuDAXnasurKGuwkNSdkJI53W5p5xnt/6b1fmrUITCub2mcVvuHcS0Vkn3jGMV8AimWaopCjAH6L5VCfx1NSz4yfXUleSjeepQHU6EamPWT54jeUD7CiJahJfWlEar5ikpZSVm38jWel5YRJjH1zzG1pIt+AOZxQcq9/PdJY+QfOA6Luo3nnN3LENp4imWdXV4V67qsNIorS/hjd2vs6l4E0qIgDspBd6q7MDf4NN05m04xg1Tcjp/cyGITY8hNr11BZDoSuS7ed9vcwxdM/DV+HDGOlDaMKWFlxrATsuGS+EndDSZaVlupdNgN2Hbu/+ipvBIY16K2U7Wy8r/+x6X/fmjiPuXLE7QPdpNWXSY4zXH2FaytVFhNCDRORK7hX9OuJ7NKQP45tLnTux0OlF79+7QdUrqS3ho8Tep0+oazT4mZuiSNFSkYaf2+GTcfoMRRXX0qvZRcrSWI24HfcZmdfoew4mUkvWvbWHzO9sxdAPVoZJ3/ShGXtmViV6bgF8DRzDfr5mYs/2uWvmMI3RgoxuY1UXXDA+Hln8UlMgIUFeST31ZIVHJ3Ts59mzibDD0npHk1+ZjU0L0JlAMbO5SvHYnywdM4GjCiYJywmYj+qsdc4i+sfu1EAoDVKHQL2YYvpI8SnfciKMuhjm7yhhWXE+iR8eRX8OC33zOlg+7R2z+xre2sentbfg9GrrfwFfrZ81LG9m1MHRk1qlzFPgmcADT1+AHPsN0SncVCcB9mNV+GmbmbmAYrTTD7DYottCmTCklQrXmtt0JS2n0UPrG9g1aZQBIQ8FfG5iVCcH23sMRbjdqVhbJL72Amt52ox/dkPib9BzYVLQpSGGA6Xyv0opJdw7CJmMYWVSHQ5PNWv1oXp01L2zA7w1f6Wnd0PG21gSpFaSUbHx7e7PQ3Ab51r/ezkBAXYP9i2DHO2YP8JPyGsGzfj+wi+ZxJeHmJuBJzPauMzBXNk/S3Y0KA86/FtXRvLSdEAoJ2YNxJ3TfDPizke79SbJoldSoNCZnTmJV/mp8hvkQlRKkYaO+2Awdtbld9P3JI6T1cqHm5LRpF671avxh3nYWbi1ANyRDMuP4wZW5JLmTOF4bOs+h1FOCI/ENLky+H9d2f8jecEIRVByubFcRwXqfhhAClz14JK/u5ZktT7P48CJ0qZMV3Yv7x3yT3JSTO0kNzcBXF7pZUF1ZcPZyEAWb4cULwe8xJ/CaD2b9CqZ+t42TGlYYLVGB40BXVlYdFfjpOQy59CaKtq2maMc6pGGg2GzYXdFMffjPkRbNogWW0uiGNES0ncz59/C47/DGrtf4cP+HVHnr8FX3pubYNAzNtJnbbQrTJg3BZjv5gvLbL61n+7FK/IFmOzuOV3Hvf9bw46/NYU/57lZn937DR1TWeoYNHcfRDflB+w3NwJ3QdnHkw6W1/PrdrWw7WgnAOTmJ/OSqkaQ16eL2h7W/ZUPRhsbV1dGaI/x85U/508y/0Ce2b5vjKzaFmOSokLWkEk9WtdXQ4aWLobao+fbPfgZ9JkPfqa2cOBr4khP5Eg346bTC8NbA+n/BznchOg0mPmh2CzwDUG0OZv7gaUr3bqF032aikjPJGjOtVbOVReSwzFPdiMLKer778pec+8tPmf6rT/nJG5uobGWGDGBTbNww7Ou8fNkr/HjUs5B/DU6ZgtuukpXg5h+3jsfRDoWxp6CaXflVjQqjAU032L0vmVtzb8OhhM7tkEiOVh9l9NzhzYodgvmwTh+SErp1aoA6r8Zdz6xmy5EKdEOiG5L1B8q465nVaIE+5MV1RXxZ9CU+o/l74Tf8vLPn7ZPenxCCSbePw9ai94PqUJl027i2Tz68DHwhopL89bDuX22c+BVMf0LT99+J6ZDuRHCAtwaeHgeLf2LKtONtePlSWPXXjo/VjUkeOJLBF3+d3nmzLIXRTelxKw0hxGzgr5jr/GeklE9EWKSw4PHp3P70KsprfRgSdGDJ9kJ2F1Tzyv1TUU7Sr2LiwBTmfW8mu/Or2H68CsOQ7C7fw2sH5nG4+hD94wfw1SE3kBOfE3TukdLakP0wfLpkT2EN915wBRMzJ3P3gjuaZYIDqEJlcOIQeo/OZNJt41j13JcIITA0g/ShKVz4yPQ25f50awE+v9GsZpEhodrjZ/nuYmYMS6egtgC7Yg/y4RjS4HD1oTbHb2DA1GzsLhtrX95EVUENiX3jmXDTGLJy2/bx4K1upVe5hDYq6JopTi8B/wesBKIws7K/3i55g1j/NFQeOVFGHcBfBwt/AGNuA1eku88ZwJvAK5ghxpOA+wEr6ulMo0cpDSGECvwDuBAzPGWtEOJ9KeX2ts/s/izcVkCdV6dp8zzNkBRXeVi9r4TJg07e0a681seP3thsrk7ch4nOeQ8hNBBQUFvAusK1/Grq4wxNal4DaEB6LFqIHtAOm8KI3qb5JsWdwux+l/DpoQWNpiqBwKE6uHrQNQDkXjKYIecPoPxIBe54V7s63x0traXeH+xo9+kGx8rNB2Tv2N4hnf42YWNw4pCTXqOBvuN60Xdcr3Yfb550LughVnv2aBh+7UlOzgQe79j1WmPXe80VRgOqA46vhf6Rjo76LTAPaEgo/QRYAbwBJEVKKIsuoKeZpyYAe6WU+6WUPuBVzFTXHs++wuqQD0+/bnCopLWkreb88p0tFFXWU+fTcWUuQShaY+SlRDY6k1uSnRLN+P7JOJuYsoQAl13lqrw+jdvuHHk3Nw+/hVR3Gm5bFOPS8/jDjD+REX1iNmlzqKQOSG53q9TBWXG4Q7QMtasKgzLMbm+JriTO6zMLp3oiOa5BYc0deHW7rtNp3Alw0R/BFmUWVARTYWSMhhHXd+21mxKdTssy6QAYGrgj/VAuAT7ghMIAc+VRjxlFZnEm0aNWGkAvzEypBo4CQTWvhRB3A3cD9O3btpO0uzAwIxa3Q6W+RctVu6qQk3ryZDCPT2fDwXLMBYOB6gpd6WV/Rei8hMe/OoZnl+zl3S+P4vXrjO+fzMOzh5IYfcKXoQiFKwbM4YoB4dPTM4em869FeymorG9c7ThUheyUaPL6nXgY3jfmfjKjs/hg/3vU+esYnpzLHSPvJDXqNJQvn/AN6DXe9GF4ymHY1ZB7HQRqfRXuKmb9q1uoOFZFcr9E8m4YRXJOmGsmTXwA9swzTVINCAXi+0DGmPBeq8PsARwEO/19mMEAFmcSPU1ptGJcbrFByqeBp8GsPdXVQoWD84dn8NTCPfj8Og2WIpsqyIh3MaH/ycNVdSmbvBECaTgQarBZpbU6TA6bwn0XDua+CztSUfXUsdsUnr1rIk8t3MPi7YUoAmaPyuLuWQObRY+pQuWawV/hmsFfOa3yNdJrvPnTgsNfHufTJz5vzAGpLqrh6IbjXPHrC8PbxCl7Glz4O1jwPVNZGbqpML4+vxWfy+kkk9bDi3NOryinGcMwOH78OH6/n8zMTFyutiMFzwR6mtI4CvRp8ro3ZtB7j8flUPnP3ZP447wdrNhTjCIEM5Ik92ib8XxYjvviixDO4NpFRl0dVb/7A3VvvkG/mQ+wL6kvUgjqikcRlboRRT3xZXaqziBzjlFdTe0LL+JZsAAlJYWY22/HOXVKl99vU+KjHDx6ZS6PXtnz2s4vf3pt86TBQMvZlf9Zz5wnLg7vxSbcD6NvhuPrTJNU+qjQCuP4elj7FNQVw9CrYOTXzC6BXUYOZtb5VponNNqBr3XhdSNLaWkp8+bNQ9fN/79hGOTl5TF6dPcrsR9OWq1y2x0RQtiA3Zg1EY4Ba4GvSSm3tXZOT6lyKzUNz8KFaHv3ofTKoubv/0A/chTp8SBcLkRMDKkfvIet1wlHrpSS4jlz8W/dCl4vRxIy+dEVj6Kpdrw2GwnZX+BI2Ibb7kCXOpf0u4zbRtzeWCbcqKmh6OJL0AsKGiviCrebuEe+T8xdd0bkfehJ6H6dZ699NWRHOtWhcucbEaj1uf7f8PHDoHnMLoj2aEgZArcvB3tXzoJrgF8CSwOvU4AfE8J6fEZgGAYvv/wy9fUtmk3ZbFx66aVkZPT8qLEOV7ntjkgpNSHENzFDM1TgP20pjJ6CXlJC8ZVzMEpKkR6POXvUTqwQZG0t0uOh4tvfIeW1Vxu3+9atQ9uxA7xmNFOfinyeeu1Rvhg2nZKLrmD0uAeYOCiaSn8Z6VHpRNmjml239qWXmykMAFlfT+UTvyXq+q+ixMZ28Z2HF79H4/jWAhRFIWtkOmqIzPJwoqgKqlNF8wSbZlyxXV3RNgTeapj/UIuw3Foo2Qmbnoe8e7rw4jHA7zCbfNZjRkxF2mzWdRQUFKBpwf93TdPYsWPHGaE0WqNHKQ0AKeVHwEddNr6uUz/vI+rffRfhchF1/fW4pk/rqssBUPHjn6AfO95MUQSh63hXrUbW1yPcZjtU//YdSMNodli0r55LNn1C1Og0EkebDuskQmc9exYsCOq5ASDsdvybNuM8t7Vs5+7HvuWH+OyvKxCBcucCuOgHM+g1quu+vEIRjLhsCFs/3NnMRGVzqoy5+vQ0PGrG0VWmv6NlaK6/Dra90cVKo4GowM/p5fjx46xcuZLy8nLcbjdjxoxh+PDhXVZS3e8PVU3YxOvtWG20nkaPUxpdiTQMSm+7A9/Klcg6M0rFs+BTom+/jfgf/qBrriklno8/aVthnDgYo7YWNaA0bDnZCFUNigQQbjf2wSd3aCspKeaqpqV5RddREntOx7TqohqW/GUFuk/HTIs0+fixz7jpv1fjiOpcp8L2MP7ro/HV+ti1eD+KKjB0ycgrh5F7WfvzR8KGIzb4f9lAxMNyu46CggI+/vjjxpl/bW0tq1evxufzMXbs2C65ZkZGBkaLCRuY5qn+/ft3yTW7Cz0tT6NL8X7+eTOFAaa5puaZZ9GOHGnjzFOkvX4lKSmcOJnqp/8NgHPaNJT0dLA10f1CgNNJ1NVtN1rSDcmmy2/itfFzWTxoCh5b4MGqqqi9emEb3pV9JsLLns8PII3g91AAB1Z14f8N00Q17b6J3Pz8Ncz93WxuefFaJtw4JjJNg3pNAHciQWYhexSMv+/0y3OaWLduXZCpSNM0Nm7c2OikDjdOp5NJkyZha/Lds9lsJCcnM2DAgC65ZnfBWmk0wfPpwmYKowGhKHi/WIrt6yePBDEMySdb8nl77RG8ms7FI7O4ZkKfkJVbwayL5LrgAjyffgon+4AbBtLjofp3v8c+aCCu884j9Z23qPjeI3gWLwYpcYw7h4Tf/x4lvvVCfLVejXueXc3xcg91oy7B5ffywsTr+M2CP9EnwUXySy90205pHs3DusK1+HQfY9PGkuhKwlfrx9CCZ32GYeCva92MEE4cUQ4cfbtuRdMuFAVu/BheuBC8VYAAwwfTfwz9zousbF1IWVnonCQpJfX19cTEdE3Tq9zcXNLS0ti+fTter5f+/fvTv39/FOXMnotbSqMJIj7enLW3NBUpCqKdTuHH3tvKom2FeALZ3YeK97Bwaz7/vnMitlbaiyY8/muKN2/GqKxE1taC09no3A6FrK+n5l//xnXeeagpKST/91mkzweGgWhHnPizn+3lUEltoEChwGN34bXDP279Nf95cGa3VRibijfx2KpfIoRASokudW4efgvjx01m2/zdIRzSgt5jMkOO1Vm8tT5K9pXhjneRlJ0Q1rHDQuow+NYhs6hhfblZBiX6zO5HkZCQQEFBQch97oApt6tITU1lxowzo9Jwe7GURhOir7uWmqf/Haw0hMB14QUnPf9gcQ0LtxbgbTLr9WoGB0tq+XxnEefnhnbKqunppC/7gvqPP0bbs5faF1/COIkzTS8ubi6io/2z3AWbC4Iq2kpgT6Wfao9GnLv7VRf1aB4eW/VLPHpzx/1LO15kxLkj6XNOFke+PN6oOGwuG0MvHEBC75OUPu8AG97axvpXN6PaFAzdID4zjkt+eh7Ryaff8dsminrGlExvD3l5ecyfP7+ZKcpmszFy5EhUtWsj6M5Gzux1VAex9etH4h9+j3C7EbGxiJgYRHw8yS+9iNLKjKX+o/kUXXoZ+XnjWfHbpxAh/BP1Pp01e0vbvLZwOomaM4fY+7+B0cpyuxGHA9esUzA3tLaQkF0XJFlUV8TGog2U1Len610w6wvXhVwB+XQfi48s4sLvTWPWw1PImdyHAdOyuejR6Uy5IyjEvNMcXn+ML1/bjO7T8dX50bw6ZYcr+OTxz8J2DYvOkZWVxQUXXEBcnFnp1+l0cs4555CXF77/v8UJrJVGC6LmXoXrogvxrVwFTifOSRMR9tAz7+p/PU317/+ADCT4RK1cipgxOCiJyq4KUuPaGbfvdCKczsYxg3A4UBISiLm38+GTl4zK4rXVh/A1WREJAUOz4ogN8yrDr/v5/brfsr5wHXbFgd/wMSlzMt8a9x1sSvs/fj7DFzKJTiLx6B6EIug3uS/9JndNrbEt7+8MahcrDUnZ4Uoq86uJz+xZOS1nGtnZ2WRnZ2MYxhnvU4g01rsbAiU6GtcF5+Oadm6rCkPW11P9hz82e7iPObwZh+YLWm2oiuDyse0ryS0UhahbboaWvglFQcnMJObee0hb9Clq8snrUbXG7TP70z8tBrdDRRUQ5VBJiHLw82vC3yL0uW3/5cvC9fgNP3VaLX7Dz+r8Vby6838dGmdM6tiQvcpdqoupWeeGS9xWqa8MzmcBM3rKW31mx+W3FyklRmkdsipy74elMLoea6XRSbSDB81olSbYDZ1fzvs9T1z6LcrjU1GEWQjwF18ZRUZC+x1y8Y8+gqyupu7NtxB2O1LTiLnrTuIe+X5YnNRuh43/3DWJtftL2ZVfRUaCmxlD03CGOYNaSsmCQx8HddzzGT4+OjCPG4ff3O6xEl2J3Jx7Ky9ufwFN92Ng4FJdjEvPY2zaOWGVOxQ5E3pTfrQSw98iSktKksJd0bYHou8rw//eTvBqYEhERgyOa3MRcWd+Ab+zjR5Ve6ozdFXtKb20lILxE8AbXEnWfu651P793/g0g4HpsSG74rUHo7ISPT8ftU8flOj29aeIJDUeP0dK60iLd5Ec48SQBnPfuxIZXIgYVai8M+f9Dl/jQOV+Fh9ehEfzMDlrCmPTzumQIi33lHO85hgZ0Zkku9u/WvPWeHnz4Y+or/SYSYTC7B0y9a7xDL2wkz2/zxCMsjp8/1oHTRWqAJHoxnH/hG4bjWfRNmdE7anuhJqcjOv88/EsWtwsPFa43cQ/8E3S2tED42Qo8fFt5lt0F6SUPLlwD6+tOoRdVfDrBtOGpPLTuSMZmDCQPRV7gs4Zlty5Mhv94vtzx8iOZ9zqhs7fN/6NL45+3tg6dlLmZB4e923sysn9OM4YJ1/5y2Vsm7+bw+uPEZ3kZuSVw8gYevKOimc6+rrjoLdcgYGs8SGPVCL6JkRELouuwTIAngKJf/sr7osvAqfDjLhKSCD+N4/3qJpN4eDddUd5I+BYr/Vq+DSDpbuK+dP8Hdw7+n5cqgtVmKYvm7Dhtrm5a+TpqIN0gtd2vcqyY0sDvpW6Rt/KC9ueb/cYzhgH51w7gqueuJgLvz/dUhgBZIXHbNQXal91iFa5Fj0aa6VxCihuN0lPPYlRVYVRXo7aqxfCdva9pS+vOICnha3fpxnM35TPdy49n7/O+jvv7X2XA5X7GZgwkDkD55J2OjruNWHe/g8ae5s3ymj4+OTgfG4fcYdlQjkFlH6JGPvKmpunwPRt9LKiys40zr4nXBegxMWhBGLEz0YqWinVYRiSep9GZnQm946ObO2jOi24PAyAV/cikYgzuIx3V6OOzkBfeQRZ7aWx7aRdQR2RjtKBABCLnoFlnuqheP06i7cV8NaawxwoqomoLGP6JoZ85CbHOrtNdvmQpKEht/eL79/YlMqicwiHiuOucagTeyOS3IiMGGyzB2G7/PS2DrY4PVgrjR7InoIq7n9uHZpuoEsJEs4fkcGP54xA6WSk1qlw/4WD+fJgGV5NRzcChXZtCo9c3nX9DDrKXSPv4QdLv4/P8GNIHUUo2BU7947+RqRFOyMQbjv2CwbABWd2hVcLK+S2xyGl5Ko/f0Fhi2Qzl13lh3NyuWhkeAv0tZejZXW8sHQ/W49W0jc5ipun9Wd4r9CRX0WVHj7YcJSCSg8T+iczc1g6dlvXz/bza/N5e8+b7KvYS05cP64e9BV6x/bu8uueqVTnH6SutJCE7ME4Y61clTON1kJuLaXRw9hTUMXdz66h3hecHT02J5GnbpsQAanaz7r9pXz3fxvQDQO/LnE7VHonRvH0nRNwO6yFb0/AV1PJF7//JmX7t6DY7Oh+P0MuuYnRX/t2t1lZWpw6rSkNy5jbw/DrslWXrT9ET4nuhGFIfvrWZjx+vbHKbr1P53BpLa+tOhRh6Szay8p/PELp3k3oPi/+uhoMv5fdn7zMwaUdT9a06HlYSqOHMTgjNmRfDpddYfao8Jmm/Dt2UHLTLRzPHUnhebOoe/fdUx7zYEltyBWSVzNYsCV0PwSL7oWvppKCzSswtOYRc7q3np0f/jdCUlmcTiyl0cOwqQq/+sooXHYFe0B5uB0qgzPjuHJcn7Bcw797N8VXXoV3yRJkRQXa7j1UfPf7VD/11CmN67ApGCHasoLpOLfo/vjqahBK6Bpl3uqK0yuMRUSwjMhhRi8qQtu9BzW7L7Y+4XmIt2TiwBRef2Aa8zYeo7TGy4QBKUwdnNrpGlctqfrjn5AeT7Pe5bK+nuo//YWY225rV3fAUPROiiIr0c3BktpmbdFddpWrx3fNe2URXqJTMrG7o9F9zQMxhKKSOabrqw1bRJ6ITO+EENcKIbYJIQwhRF6LfT8QQuwVQuwSQlzcZPs4IcSWwL6/iW7mcZO6Tvl3v0fBpMmU3nUXhTPOo+TmWzBa64txiqTFu7htxgC+e9lwpg9NC5vCAPBv2ABGCP+IEGjHjp/S2L+9fixJ0Q6iHCouu4LTpjBjWBqXjWlf6XiLyCIUhfF3/xLV4YJAfotic+CIjmPkV74ZYeksTgeRWmlsBa4G/tV0oxBiOHA9kAtkAQuFEIOllDrwFHA3sAr4CJgNzD+dQrdFzb+epu7d98DrQwYq33qXLafyxz8h8Y9/iLB0HUPNzkYPoRykpqGmnVq9pb4p0bz37Rms3ldKSbWXUX0T6BeG4o4Wp4/eebO44Jcvs/PD56gpPEJ67gQGX3IT7oQzuxe5hUlElIaUcgcQKjxvDvCqlNILHBBC7AUmCCEOAnFSypWB814ArqI7KY1n/wMtVxVeL3Vvv0PCb5/oUTWpYh98kLIvN5gmqgZcLqLmXoUSe+q1hGyqwtTBVrG/k6MBrwLvBv6eDdwMRL4neVK/4Ux54HeRFsMiAnQ372Mv4EiT10cD23oF/m65PSRCiLuFEOuEEOuKi4u7RNCWyOrq0Dt0HenrWZU+XdPOJeFPf0RJSQGnA1wuoq+7loTHH4u0aGcZ3wX+CRzE/Mi/ANyOqUAsLCJDl01/hRALgYwQu34kpXyvtdNCbJNtbA+JlPJp4Gkwk/tOImpYcEyahHfx4mbOYwBb/34oUZGfGXaUqDlX4r7icoyyMpSYmE47vy06y3ZgHdDU4ewDjgOfARdEQCYLiy5UGlLKznyqjwJNw2h6Y35Ljgb+brm92xD/059QvGaNadLx+0FVEQ4HCU/8JtKidRqhKKgplp06MmwldJOKOmAjltKwiBTdzTz1PnC9EMIphOgHDALWSCnzgWohxKRA1NTNQGurlYhgHziA9MWLiL71VuzjziHquutI/fgjnJMmRVq0dqOXlVP1hz9SdNkVlN55F97VqyMt0llMGqHndE5CL+AtLE4PEfHOCiHmAv8HpALzhBAbpZQXSym3CSFex1yba8D9gcgpgPuA5wA3pgO82zjBG1CzMkn4+U8jLUan0MvKKLrwYozycvB68W8E72efE/+rXxB9ww2RFq/74a2G7W9BbSH0nQZ9JpvlfcPGVMyPej3NLbEqcFkYr2Nh0TGsgoUWAFT+5glqnv43tHDai+hoMjdvtHwaTTm2Dl64AAwNdC+oTsiZCde/C2o452FHgEeBA5huvVTg18CIMF7DwiI0rRUs7DlxoBZdimfRoiCFAYAQ+HfvxjFq1OkXqjsiJbx+DXgrT2wzNDi4BDY8C3nh7H3eB3gZKMJceGcSOibEwuL00d18GhYRQkkO7fCWmoaSaPVKaKRoG9SVBm/318GX/+mii6Zh5rpaCsMi8lhKwwKA2HvuRrhb9HO22XCMGNFlNbR6JrIN30X3Lk1vYREOLKVhAYBr1nnEfufb4HIhYmMRbjf2kSNIevbfkRate5GaC8644O32KBhz2+mXx8LiNGM5wi2aYdTU4N+2DSUlFfuA/pEWp3tyZCW8eDFI3TRLOWKg10S4cT6o9khLZ2ERFixHuEW7UGJicE6cGGkxujd9JsPDB2Hba1BTCNnToN+sMIfcWlh0TyylYWHRGaKSYPx9kZbCwuK0YykNix5LxdFKti/YS315PX3zetF/Sl9Ue+iuchYWFuHBUhoWPZJ9yw/x2V9XoGsGUpccXHOULR/s5MrHL8LmsBSHhUVXYUVPWfQ4NJ/O5/+3Es2rI3UzkEPzaJQdrmDnp3siLJ2FxZmNpTQsehzFe0sJleime3X2LT10+gWysDiLsJSGRY/D5lRpLVTc7rZCXi0suhJLaVj0OFL6J+GOcwYtNmxOleGzB0VGKAuLswRLaVj0OIQQXPKT83DHu7C7bdhdNlS7yvBLhpA9offJB7CwsOg0VvSURY8ksW8CN/7nao5tKsBT5SUzN42Y1OhIi2VhccZjKQ2LHouiKvQ5JyvSYlhYnFVY5ikLCwsLi3ZjKQ0LCwsLi3ZjKQ0LCwsLi3Zj+TQsupSj1UfYWLyRaHs0kzIn47a5T36ShYVFt8VSGhZdgpSSf256kkWHFwKgCIV/bnqSn03+JcOTh0dYOgsLi85imacsuoR1hWtZcmQxPsOHz/Dh0T3Ua/U8tuqX6IYeafEsLCw6SUSUhhDi90KInUKIzUKId4QQCU32/UAIsVcIsUsIcXGT7eOEEFsC+/4mhNXxpjuz4NAneHRP0HZNauwo2x4BiSwsLMJBpFYanwIjpJSjgN3ADwCEEMOB64FcYDbwpBCioc71U8DdwKDAz+zTLbRF+9EMLeR2gWh1n4WFRfcnIkpDSrlAStnw5FgFNNR+mAO8KqX0SikPAHuBCUKITCBOSrlSmpXqXgCuOt1yW7SfmX3Ow6W6grZLJMOTcyMgkYWFRTjoDj6N24H5gb97AUea7Dsa2NYr8HfL7SERQtwthFgnhFhXXFwcZnHPbAor61m7v5SiymDTUkc4t9c0RqaOalQcNsWGQ3HwrXHfwaE6wiGqhYVFBOiy6CkhxEIgI8SuH0kp3wsc8yNAA15uOC3E8bKN7SGRUj4NPA2Ql5fX6nEWJ/BpBj97azPLdxfjUBV8usHMYWn8dO5IbGrH5xaqUPnxxJ+yuWQT6wvXE2uPYWafWaRGpXaB9BYWFqeLLlMaUsoL2tovhLgFuBw4X55ojnAU6NPksN7A8cD23iG2W4SJpxbuZsXuYnyagU8zAPh8RxH/TtjLfRcM7tSYQghGp45hdOqYMEpqYWERSSIVPTUbeAS4UkpZ12TX+8D1QginEKIfpsN7jZQyH6gWQkwKRE3dDLx32gU/g3l33VG8AWXRgFczeHvtkVbOsLCwOBuJVHLf3wEn8GkgcnaVlPJeKeU2IcTrwHZMs9X9UsqGoP77gOcAN6YPZH7QqBadQkqJxx86d6LOZ+VUWFhYnCAiSkNKObCNfY8Bj4XYvg4Y0ZVyna0IIRjWK57txyqD9o3sk3D6BbKwsOi2dIfoKYsuRC8pofrJpyj//iPUvfkW0hM6Kup7lw3DbVdRFTPmwKYI3A6V71w69HSKa2Fh0c0RJ3zQZyZ5eXly3bp1kRYjIvg2baLkuuuRmgYeDyIqCiU9nbQP30dJSAg6/mhZHa+sOMiugiqGZcZzw5RsshKjTr/gFhYWEUcIsV5KmRe03VIaZy6FM2ai7d3XfKPDQfStt5Dws59GRigLC4seQWtKwzJPnaHohYVoR0JEPvl81H/w4ekXyMLC4ozAUhpnKnYHtLKKFA77aRbGwsLiTMFSGmcoalIijtFjQFWb73C5iP7a1yIik4WFRc/HUhpnMIn/+D/UjAxETDS4XQi3G+eUycTcfVekRbOwsOihWJ37zmBsvXqRvnI53s+/QD9+HPvoUThGjoy0WBYWFj0YS2mc4QhVxTXrvEiLYWFhcYZgmacsLCwsLNqNpTQsLCwsLNqNpTQsLCwsLNqNpTQsLCwsLNqNpTQsLCwsLNrNGV97SghRDByKtBydIAUoibQQp4h1D92HM+E+rHs4vWRLKYP6M5/xSqOnIoRYF6pYWE/Cuofuw5lwH9Y9dA8s85SFhYWFRbuxlIaFhYWFRbuxlEb35elICxAGrHvoPpwJ92HdQzfA8mlYWFhYWLQba6VhYWFhYdFuLKVhYWFhYdFuLKXRzRBCzBZC7BJC7BVCPBppeTqDEKKPEGKJEGKHEGKbEOKhSMvUWYQQqhBigxCiR/bIFUIkCCHeFELsDPw/Jkdapo4ihPhW4HO0VQjxihDCFWmZ2oMQ4j9CiCIhxNYm25KEEJ8KIfYEfidGUsbOYCmNboQQQgX+AVwCDAduEEIMj6xUnUIDviOlHAZMAu7vofcB8BCwI9JCnAJ/BT6WUg4FRtPD7kUI0Qt4EMiTUo4AVOD6yErVbp4DZrfY9iiwSEo5CFgUeN2jsJRG92ICsFdKuV9K6QNeBeZEWKYOI6XMl1J+Gfi7GvNB1SuyUnUcIURv4DLgmUjL0hmEEHHAdOBZACmlT0pZEVGhOocNcAshbEAUcDzC8rQLKeUXQFmLzXOA5wN/Pw9cdTplCgeW0uhe9AKONHl9lB74sG2KECIHGAusjrAoneEvwPcBI8JydJb+QDHw34CJ7RkhRHSkheoIUspjwB+Aw0A+UCmlXBBZqU6JdCllPpiTKyAtwvJ0GEtpdC9EiG09NiZaCBEDvAU8LKWsirQ8HUEIcTlQJKVcH2lZTgEbcA7wlJRyLFBLDzOHBGz+c4B+QBYQLYS4MbJSnd1YSqN7cRTo0+R1b3rIUrwlQgg7psJ4WUr5dqTl6QRTgSuFEP/f3v2ExlHGYRz/PtiKfyqoxINQpFLEgzlU0oLWi7YSb1IwPYlspXgQ7M0eVETwDyiCF++lB0XRWLGKEBdKeylIrCZq1SJq0YD/oB5UPBh5eph3ybrG7cymMkl9PrBkZ3dm5wcJ/PZ9Z/K8p6mmCXdIeqndkhpbABZs90Z501RNZC25E/jG9s+2/wQOAdtbrmklfpR0LUD5+VPL9TSWprG6zAI3SLpe0sVUF/wOt1xTY5JENY/+ue0X2q5nFLYfsb3R9iaq38MR22vqG67tH4DvJN1YXtoJfNZiSaP4FrhF0mXl72ona+xi/oDDQKc87wBvtVjLSNa1XUAssb0o6SFghuoukQO2T7Zc1ihuA+4DPpE0V1571Pa77ZX0v7UPeLl8CfkauL/lehqx/b6kaeBDqrvyPmKNRHFIegW4HRiTtAA8ATwLvCZpL1VD3N1ehaNJjEhERNSW6amIiKgtTSMiImpL04iIiNrSNCIiorY0jYiIqC1NI2IISX9Jmut7bBrhM3b9l4GNkjolNfVLSZ1zHxExutxyGzGEpN9sb1jhZxwE3rE93eCYdbYXa+x3NfABsJUqcuYEMGH7lxHLjRgqI42IhiRNSDom6YSkmb5YiAckzUqal/RG+S/m7cDdwPNlpLJZ0lFJW8sxYyWqBEl7JL0u6W3gPUmXlzUZZkvg4HKJx3cBXdtnSqPo8s847ojzJk0jYrhL+6am3iyZWi8CU7YngAPAM2XfQ7a32e6tW7HX9nGq6Ij9trfY/uoc57sV6NjeATxGFV+yDbiDqvEMptRecMnIsbolRiRiuD9sb+ltSBoHxoFuFYXERVSR3QDjkp4GrgQ2UMXBNNW13VuDYZIqNPHhsn0JcB1/z166oJKRY/VL04hoRsBJ28stm3oQ2GV7XtIeqtyh5SyyNMofXLr094Fz3WP71JB6FgbOsxE4OmT/iBXJ9FREM6eAa3prbUtaL+mm8t4VwPdlCuvevmN+Le/1nAYmyvOpIeeaAfaVdFck3fwv+0xKuqqsPTHJaCOciFrSNCIaKMvwTgHPSZoH5lha3+FxqhUKu8AXfYe9CuwvF7M3U61E96Ck48DYkNM9BawHPpb0adkerOdMeX22PJ7sm96KOO9yy21ERNSWkUZERNSWphEREbWlaURERG1pGhERUVuaRkRE1JamERERtaVpREREbWcBY2VqAAA4dagAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"fig, ax = plt.subplots()\n",
"ax.scatter(X2[::5, 0], y2[::5], c=np.arange(0, len(X2), 5) // 100, cmap=\"Set1\",\n",
" label=\"Partition\")\n",
"ax.set(xlabel=\"Feature 0\", ylabel=\"target\", title=\"Non-stationary data (by partition)\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's fit two estimators:\n",
"\n",
"1. One `BlockwiseVotingRegressor` on the entire dataset (which fits a `LinearRegression` on each partition)\n",
"2. One `LinearRegression` on a sample from the entire dataset"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:09.721035Z",
"iopub.status.busy": "2021-01-14T10:56:09.720328Z",
"iopub.status.idle": "2021-01-14T10:56:09.798799Z",
"shell.execute_reply": "2021-01-14T10:56:09.798408Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LinearRegression()\n",
"clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n",
" subestimator,\n",
")\n",
"clf.fit(X2, y2)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:09.807220Z",
"iopub.status.busy": "2021-01-14T10:56:09.802029Z",
"iopub.status.idle": "2021-01-14T10:56:09.861143Z",
"shell.execute_reply": "2021-01-14T10:56:09.860767Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"LinearRegression()"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_sampled, y_sampled = dask.compute(X2[::10], y2[::10])\n",
"\n",
"subestimator.fit(X_sampled, y_sampled)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Comparing the scores, we find that the sampled dataset performs much better, despite training on less data."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:09.868762Z",
"iopub.status.busy": "2021-01-14T10:56:09.863148Z",
"iopub.status.idle": "2021-01-14T10:56:10.056999Z",
"shell.execute_reply": "2021-01-14T10:56:10.056636Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"-11.096587887414355"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.score(X2, y2)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:56:10.067800Z",
"iopub.status.busy": "2021-01-14T10:56:10.060993Z",
"iopub.status.idle": "2021-01-14T10:56:10.135967Z",
"shell.execute_reply": "2021-01-14T10:56:10.136322Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.09051854628005862"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"subestimator.score(X2, y2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This shows that ensuring your needs to be relatively uniform across partitions. Even including the standard controls to normalize whatever underlying force is generating the non-stationary data (e.g. a time trend compontent or differencing timeseries data, dummy variables for geographic regions, etc) is not sufficient when your dataset is partioned by the non-uniform variable. You would still need to either shuffle your data prior to fitting, or just sample and fit the sub-estimator on the sub-sample that fits in memory."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}