{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Blockwise Ensemble Methods\n",
"\n",
"Dask-ML provides some [ensemble methods](https://ml.dask.org/modules/api.html#module-dask_ml.ensemble) that are tailored to `dask.array`'s and `dask.dataframe`'s blocked structure. The basic idea is to fit a copy of some sub-estimator to each block (or partition) of the dask Array or DataFrame. Becuase each block fits in memory, the sub-estimator only needs to handle in-memory data structures like a NumPy array or pandas DataFrame. It also will be relatively fast, since each block fits in memory and we won't need to move large amounts of data between workers on a cluster. We end up with an ensemble of models: one per block in the training dataset.\n",
"\n",
"At prediction time, we combine the results from all the models in the ensemble. For regression problems, this means averaging the predictions from each sub-estimator. For classification problems, each sub-estimator votes and the results are combined. See https://scikit-learn.org/stable/modules/ensemble.html#voting-classifier for details on how they can be combeind. See https://scikit-learn.org/stable/modules/ensemble.html for a general overview of why averaging ensemble methods can be useful.\n",
"\n",
"It's crucially important that the distribution of values in your dataset be relatively uniform across partitions. Otherwise the parameters learned on any given partition of the data will be poor for the dataset as a whole. This will be shown in detail later."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's randomly generate an example dataset. In practice, you would load the data from storage. We'll create a `dask.array` with 10 blocks."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:18:56.581069Z",
"iopub.status.busy": "2021-07-26T20:18:56.580467Z",
"iopub.status.idle": "2021-07-26T20:19:00.219135Z",
"shell.execute_reply": "2021-07-26T20:19:00.218317Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
160.00 MB
16.00 MB
\n",
"
Shape
(1000000, 20)
(100000, 20)
\n",
"
Count
10 Tasks
10 Chunks
\n",
"
Type
float64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from distributed import Client\n",
"import dask_ml.datasets\n",
"import dask_ml.ensemble\n",
"\n",
"client = Client(n_workers=4, threads_per_worker=1)\n",
"\n",
"X, y = dask_ml.datasets.make_classification(n_samples=1_000_000,\n",
" n_informative=10,\n",
" shift=2, scale=2,\n",
" chunks=100_000)\n",
"X"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Classification\n",
"\n",
"The `sub-estimator` should be an instantiated scikit-learn-API compatible estimator (anything that implements the `fit` / `predict` API, including pipelines). It only needs to handle in-memory datasets. We'll use `sklearn.linear_model.RidgeClassifier`.\n",
"\n",
"To get the output shapes right, we require that you provide the `classes` for classification problems, either when creating the estimator or in `.fit` if the sub-estimator also requires the classes."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:00.223680Z",
"iopub.status.busy": "2021-07-26T20:19:00.223135Z",
"iopub.status.idle": "2021-07-26T20:19:00.229684Z",
"shell.execute_reply": "2021-07-26T20:19:00.229307Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"BlockwiseVotingClassifier(classes=[0, 1],\n",
" estimator=RidgeClassifier(random_state=0))"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sklearn.linear_model\n",
"\n",
"subestimator = sklearn.linear_model.RidgeClassifier(random_state=0)\n",
"clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n",
" subestimator,\n",
" classes=[0, 1]\n",
")\n",
"clf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can train normally. This will *independently* fit a clone of `subestimator` on each partition of `X` and `y`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:00.254152Z",
"iopub.status.busy": "2021-07-26T20:19:00.249130Z",
"iopub.status.idle": "2021-07-26T20:19:03.015236Z",
"shell.execute_reply": "2021-07-26T20:19:03.016163Z"
}
},
"outputs": [],
"source": [
"clf.fit(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All of the fitted estimators are available at `.estimators_`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:03.022409Z",
"iopub.status.busy": "2021-07-26T20:19:03.021946Z",
"iopub.status.idle": "2021-07-26T20:19:03.027084Z",
"shell.execute_reply": "2021-07-26T20:19:03.027750Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0),\n",
" RidgeClassifier(random_state=0)]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.estimators_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These are different estimators! They've been trained on separate batches of data and have learned different parameters. We can plot the difference in the learned `coef_` of the first two models to visualize this."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:03.031591Z",
"iopub.status.busy": "2021-07-26T20:19:03.030681Z",
"iopub.status.idle": "2021-07-26T20:19:03.284329Z",
"shell.execute_reply": "2021-07-26T20:19:03.285020Z"
}
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:03.292970Z",
"iopub.status.busy": "2021-07-26T20:19:03.292490Z",
"iopub.status.idle": "2021-07-26T20:19:03.409871Z",
"shell.execute_reply": "2021-07-26T20:19:03.410521Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEFCAYAAAAWrxseAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVRElEQVR4nO3dfbBlVX3m8e9DA+JbbA2tNC9NE+xYNjhxtAV1JjVERAGDTY1YJZkZkKiMliRVU2poh3FiMlppxxgjkZJhMviaCkWSMbbSDiKG1KhFAoyiaZXQIkhDy1vwBdGQlt/8sVfHw/Wse8/tc7qb7v5+qk7dvfdaa++1zzn3PHvtvc+9qSokSRpnv93dAUnSo5chIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNiH5bk4iRvG5l/Q5K7kjyQ5OeT/KskN7f503djV3dYkk1JTtjd/dhVklyT5LW7YbtJ8sEk9yf527Zs7vvpgSS/sMB6VrR6S3ZNz7WQ+D2JvVOSW4GnAduAnwBfAz4CXFJVD4+pfwDwfeD5VXVjW3Y1sKGq3rer+v1okqSAVVW1eXf3ZVJJrgE+VlV/3Cn/ReCdwK8ABwC3AR8C3ldVP5liu78M/CnwjKr64bj306620HOhyTiS2LudVlVPBI4E1gPnA/+rU/dpwEHAppFlR86Zn1iS/Xek3b6kHX3vst/BJEcDfwPcDjyrqp4EvBJYAzxxytUfCdxaVT9s8+PeT9oTVZWPvfAB3Aq8eM6y44CHgWPb/IeAdwC/CPwQKOAB4HPAN1vdH7VljwGexBAyW4E7WtslbV2vBr4AvBf4h1b2GOD3gW8DdwEXA49t9U8AtgBvAu5u6zxnpK+PBd7DcKT7PeDzI22fD3wR+C5wI3DCJM8D8HbgcoYR1Q8YPsDWzNO2gKePWT7ffj0Z+BRwD3B/mz58pO01DEfyX2jP7dPbdl4P3NzaXEQb5bc2vw58vZVdCRw5UnYS8I32HL0f+GvgtZ39+RhwxQLvm5e35+W7ra/PHCk7FPiLtm/fAn6zLX8N8GOGEesDDCOKR7yf5j6fvdcXWNnq7d/qLfSe+3x7Le5vfTqllb2z9efHrQ/vB8Lw/ry7bfMrtN8FH/O8J3Z3B3zspBd2TEi05d8G3tCmPwS8o00/4pdz3DqAvwT+B/B44KnA3wL/sZW9muHU1m8A+7df+D8ENgBPYThS/STwe63+Ca3+7zKc9jgVeBB4ciu/qH1IHQYsAV7I8OF8GHBfq78fw4fkfcCyhZ4HhpD4cWu7BPg94Np5nsNeSMy3Xz8PvAJ4XCv7M+AvR9pe016DY9rzdEDbzqeApcAKhg/hk1v904HNwDNb/f8CfLGVHcxwSueMtp7/1J7TXkh8h5EgHlO+/WDhpLa+32rbPrA91zcA/7XN/wJwC/DSkdf/8yPrWsnPvp9GQ6L3+j6iHQu/5/4JeF1bxxuAO/npafRrRp8L4KVtH5YyBMYzgeW7+3f10f7Y7R3wsZNe2H5IXAtc0KY/xIQhwXD64B9pR8xt2ZnAX7XpVwPfHilL+8A5emTZC4BvtekTGI6kR7d3N8MoYb9W9ktj+n8+8NE5y64Ezl7oeWAIic+OlK0GfjTPc/gzIbHQfo1Zx7OB+0fmrwF+d8x2/vXI/OXAujb9aeA1I2X7MYTpkcBZjIRc69sW+iHxT7Tw6ZS/Dbh8zrbuaK/V8aOvbyt/K/DBkdd/opBY4PX953YTvuc2j5Q9rrU9ZOS5Hg2JFwF/v/09trN/B/eWh+eN9z2HMZwOWqwjGY4utybZvmw/hvPb241OL2P4pb1hpH4Yjvi2u6+qto3MPwg8geEI+SCGU17j+vHKJKeNLDsA+KsJ9+M7c7Z3UJL95/RjPvPuV5LHMZzSOJnh1BPAE5MsqZ9eGB59nnr9ekKbPhJ4X5L3jJSH4XU8dHRdVVVJxq17u/uA5fOUH8pw+mf7+h5u6zuMIWAOTfLdkfpLgP87z/p65nt9R03ynvvn562qHmz1nsAYVfW5JO9nGMWsSPJx4M1V9f0d2Id9hiGxD0nyPIZf+M/vQPPbGY7qDp7nA7VGpu9lOFo8pqruWOS27mU4LXQ0wzWHuf34aFW9bpHrnJWF9utNwDOA46vqO0meDXyJ4YN9uxrTrud24J1V9SdzC5KsAo4Ymc/o/BifZTgV9sFO+Z3As8as7w6G1/5bVbVqEX3vme/1HTXJe24+P/M8V9WFwIVJnsowYnsLwwhKHd7dtA9I8nNJfhW4jOGWwK8udh1VtRX4DPCetr79khyd5N906j8M/E/gve0XkiSHJXnpBNt6GLgU+IMkhyZZkuQFSR7DcPH1tCQvbcsPSnJCksMXu08TOrBt46AkBzF82M+3X09kCJHvJnkK8NtTbv9i4K1JjmnbelKSV7ayK4BjkvzbdjfZbwKHzLOu3wZemOTdSQ5p63t6ko8lWcrwofmyJCe2W1jfxPAh/UWGawHfT3J+kse25/7YduCxKAu8vqP1FvWeG+MuhmsntH19XpLj2779kJ9ebNc8DIm92yeT/IDhiOwC4A+Ac6ZY31kMFy2/xnA3yZ8z/+mL8xkufF6b5PsMR7LPmHBbbwa+ClzHcHrsXQznkW8H1gL/meEC7+0MR4M76728ieFDf/vjHObfrz9kuGh/L8P1n/8zzcar6uMM+35Z29bfAae0snsZbmFdz3AqaRXDXVO9dX2T4frJSmBTku8x3K10PfCDqroJ+PfAH7X+n8ZwG/VD7VTZaQzXWL7Vyv+Y4e6jHTH29R1Tb7HvuVHvA85oX/C7EPg5hoC/n+G02n0Md0ZpHn6ZTpLU5UhCktRlSEiSugwJSVKXISFJ6tqrvidx8MEH18qVK3d3NyRpj3LDDTfcW1XLxpXtVSGxcuVKrr/++t3dDUnaoyS5rVfm6SZJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSuvaqL9NJ0t5i5borFlX/1vUv2yn9cCQhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktQ1k5BIcnKSm5JsTrJuTHmSXNjKv5LkOYto++YkleTgWfRVkjS5qUMiyRLgIuAUYDVwZpLVc6qdAqxqj3OBD0zSNskRwEnAt6ftpyRp8WYxkjgO2FxVt1TVQ8BlwNo5ddYCH6nBtcDSJMsnaPte4LeAmkE/JUmLNIuQOAy4fWR+S1s2SZ1u2yQvB+6oqhtn0EdJ0g6Yxb8vzZhlc4/8e3XGLk/yOOAC4CULbjw5l+EUFitWrFiouiRpEWYxktgCHDEyfzhw54R1esuPBo4Cbkxya1v+/5IcMnfjVXVJVa2pqjXLli2bclckSaNmERLXAauSHJXkQOBVwIY5dTYAZ7W7nJ4PfK+qtvbaVtVXq+qpVbWyqlYyhMlzquo7M+ivJGlCU59uqqptSc4DrgSWAJdW1aYkr2/lFwMbgVOBzcCDwDnztZ22T5Kk2ZjFNQmqaiNDEIwuu3hkuoA3Ttp2TJ2V0/dSkrRYfuNaktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXTP5K7CSHv1WrrtiUfVvXf+yndQT7UkcSUiSuhxJ7AU8QpS0sziSkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV0zCYkkJye5KcnmJOvGlCfJha38K0mes1DbJO9O8o1W/+NJls6ir5KkyU0dEkmWABcBpwCrgTOTrJ5T7RRgVXucC3xggrZXAcdW1b8A/h5467R9lSQtzixGEscBm6vqlqp6CLgMWDunzlrgIzW4FliaZPl8bavqM1W1rbW/Fjh8Bn2VJC3CLELiMOD2kfktbdkkdSZpC/DrwKfHbTzJuUmuT3L9Pffcs8iuS5LmM4uQyJhlNWGdBdsmuQDYBvzJuI1X1SVVtaaq1ixbtmyC7kqSJrX/DNaxBThiZP5w4M4J6xw4X9skZwO/CpxYVXODR5K0k81iJHEdsCrJUUkOBF4FbJhTZwNwVrvL6fnA96pq63xtk5wMnA+8vKoenEE/JUmLNPVIoqq2JTkPuBJYAlxaVZuSvL6VXwxsBE4FNgMPAufM17at+v3AY4CrkgBcW1Wvn7a/kqTJzeJ0E1W1kSEIRpddPDJdwBsnbduWP30WfZMk7Ti/cS1J6jIkJEldhoQkqcuQkCR1zeTCtbSrrVx3xaLq37r+ZTupJ9LezZDQbuMHvfTo5+kmSVKXIwlJ2gkWO1KGR+do2ZGEJKnLkYSkvdbecjS/OzmSkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnq8k+Fa5/kv06VJuNIQpLU5UhC2oM4AtKu5khCktRlSEiSugwJSVKXISFJ6ppJSCQ5OclNSTYnWTemPEkubOVfSfKchdomeUqSq5Lc3H4+eRZ9lSRNbuqQSLIEuAg4BVgNnJlk9ZxqpwCr2uNc4AMTtF0HXF1Vq4Cr27wkaReaxS2wxwGbq+oWgCSXAWuBr43UWQt8pKoKuDbJ0iTLgZXztF0LnNDafxi4Bjh/Bv3VjCz2dkzwlkxpT5Phc3uKFSRnACdX1Wvb/H8Ajq+q80bqfApYX1Wfb/NXM3zgr+y1TfLdqlo6so77q+pnTjklOZdhdMKKFSuee9ttt+3wvkxzD/qubDu3/TS8737x9tTXehrT9nt3/W5Na1/5/UhyQ1WtGVc2i2sSGbNsbvL06kzSdl5VdUlVramqNcuWLVtMU0nSAmZxumkLcMTI/OHAnRPWOXCetnclWV5VW9upqbtn0FfNsace+UjaNWYxkrgOWJXkqCQHAq8CNsypswE4q93l9Hzge1W1dYG2G4Cz2/TZwCdm0FdJ0iJMPZKoqm1JzgOuBJYAl1bVpiSvb+UXAxuBU4HNwIPAOfO1bateD1ye5DXAt4FXTttXSdLizOQP/FXVRoYgGF128ch0AW+ctG1bfh9w4iz692jnKR9Jj1Z+41qS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpa6qQSPKUJFclubn9fHKn3slJbkqyOcm6hdonOSnJDUm+2n6+aJp+SpJ2zLQjiXXA1VW1Cri6zT9CkiXARcApwGrgzCSrF2h/L3BaVT0LOBv46JT9lCTtgGlDYi3w4Tb9YeD0MXWOAzZX1S1V9RBwWWvXbV9VX6qqO9vyTcBBSR4zZV8lSYu0/5Ttn1ZVWwGqamuSp46pcxhw+8j8FuD4RbR/BfClqvrHcR1Ici5wLsCKFSt2bC+kXeTW9S/b3V2QFmXBkEjyWeCQMUUXTLiNjFlWEzVMjgHeBbykV6eqLgEuAVizZs1E65UkTWbBkKiqF/fKktyVZHkbBSwH7h5TbQtwxMj84cD2U0nd9kkOBz4OnFVV35xgXyRJMzbt6aYNDBeW17efnxhT5zpgVZKjgDuAVwG/Nl/7JEuBK4C3VtUXpuyjpD2Yp+h2r2kvXK8HTkpyM3BSmyfJoUk2AlTVNuA84Erg68DlVbVpvvat/tOBtyX5cnuMu14hSdqJphpJVNV9wIljlt8JnDoyvxHYuIj27wDeMU3fJEnT8xvXkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV3Tfplur+KXdiTpkRxJSJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktTll+mkRfJLl9qXOJKQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeryT4XPiH8+Wnsz39/7rqlGEkmekuSqJDe3n0/u1Ds5yU1JNidZN2n7JCuSPJDkzdP0U5K0Y6Y93bQOuLqqVgFXt/lHSLIEuAg4BVgNnJlk9YTt3wt8eso+SpJ20LQhsRb4cJv+MHD6mDrHAZur6paqegi4rLWbt32S04FbgE1T9lGStIOmDYmnVdVWgPbzqWPqHAbcPjK/pS3rtk/yeOB84HcW6kCSc5Ncn+T6e+65Z4d3RJL0sxa8cJ3ks8AhY4oumHAbGbOsFmjzO8B7q+qBZFzzkRVVXQJcArBmzZqF1itJWoQFQ6KqXtwrS3JXkuVVtTXJcuDuMdW2AEeMzB8O3Nmme+2PB85I8t+BpcDDSX5cVe9feJckSbMy7S2wG4CzgfXt5yfG1LkOWJXkKOAO4FXAr83Xvqp+eXvjJG8HHjAgpD2Xt9Duuaa9JrEeOCnJzcBJbZ4khybZCFBV24DzgCuBrwOXV9Wm+dpLkh4dphpJVNV9wIljlt8JnDoyvxHYOGn7OXXePk0fJUk7zj/LIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkrqm/c90krTX8j/qOZKQJM3DkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpK1W1u/swM0nuAW7b3f2QpD3MkVW1bFzBXhUSkqTZ8nSTJKnLkJAkdRkSkqQuQ0JaQJKfJPnyyGPlDqzj9CSrd0L3pJ3K/ychLexHVfXsKddxOvAp4GuTNkiyf1Vtm3K70lQcSUg7IMlzk/x1khuSXJlkeVv+uiTXJbkxyV8keVySFwIvB97dRiJHJ7kmyZrW5uAkt7bpVyf5sySfBD6T5PFJLm3r/FKStbtrn7VvMiSkhT125FTTx5McAPwRcEZVPRe4FHhnq/u/q+p5VfVLwNeB11TVF4ENwFuq6tlV9c0FtvcC4OyqehFwAfC5qnoe8CsMQfP4nbCP0liebpIW9ojTTUmOBY4FrkoCsATY2oqPTfIOYCnwBODKHdjeVVX1D236JcDLk7y5zR8ErGAIIGmnMySkxQuwqapeMKbsQ8DpVXVjklcDJ3TWsY2fjuQPmlP2wznbekVV3bTDvZWm4OkmafFuApYleQFAkgOSHNPKnghsbaek/t1Imx+0su1uBZ7bps+YZ1tXAr+RNmRJ8i+n7740OUNCWqSqeojhg/1dSW4Evgy8sBW/Dfgb4CrgGyPNLgPe0i4+Hw38PvCGJF8EDp5nc/8NOAD4SpK/a/PSLuPfbpIkdTmSkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXf8f2j1ET8Pz620AAAAASUVORK5CYII=\n",
"text/plain": [
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"a = clf.estimators_[0].coef_\n",
"b = clf.estimators_[1].coef_\n",
"\n",
"fig, ax = plt.subplots()\n",
"ax.bar(np.arange(a.shape[1]), (a - b).ravel())\n",
"ax.set(xticks=[], xlabel=\"Feature\", title=\"Difference in Learned Coefficients\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That said, the assumption backing this entire process is that the distribution of the data is relatively uniform across partitions. The parameters learned by the each member of the ensemble should be relatively similar, and so will give relatively similar predictions when applied to the same data.\n",
"\n",
"When you `predict`, the result will have the same chunking pattern as the input array you're predicting for (which need not match the partitioning of the training data)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:03.417979Z",
"iopub.status.busy": "2021-07-26T20:19:03.417537Z",
"iopub.status.idle": "2021-07-26T20:19:03.425853Z",
"shell.execute_reply": "2021-07-26T20:19:03.425431Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
8.00 MB
800.00 kB
\n",
"
Shape
(1000000,)
(100000,)
\n",
"
Count
31 Tasks
10 Chunks
\n",
"
Type
int64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array<_vote_block, shape=(1000000,), dtype=int64, chunksize=(100000,), chunktype=numpy.ndarray>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds = clf.predict(X)\n",
"preds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This generates a set of tasks that\n",
"\n",
"1. Calls `subestimator.predict(chunk)` for each subestimator (10 in our case)\n",
"2. Concatenates those predictions together\n",
"3. Somehow averages the predictions to a single overall prediction\n",
"\n",
"We used the default `voting=\"hard\"` strategy, which means we just choose the class that had the higest number of votes. If the first two sub-estimators picked class `0` and the other eight picked class `1` for the first row, the final prediction for that row will be class `1`."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:03.429224Z",
"iopub.status.busy": "2021-07-26T20:19:03.428760Z",
"iopub.status.idle": "2021-07-26T20:19:04.311265Z",
"shell.execute_reply": "2021-07-26T20:19:04.310859Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([0, 0, 0, 1, 1, 0, 1, 1, 1, 0])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"preds[:10].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With `voting=\"soft\"` we have access to `predict_proba`, as long as the subestimator has a `predict_proba` method. These subestimators should be well-calibrated for the predictions to be meaningful. See [probability calibration](https://scikit-learn.org/stable/modules/calibration.html#calibration) for more."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:04.329537Z",
"iopub.status.busy": "2021-07-26T20:19:04.317513Z",
"iopub.status.idle": "2021-07-26T20:19:06.746270Z",
"shell.execute_reply": "2021-07-26T20:19:06.745507Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LogisticRegression(random_state=0)\n",
"clf = dask_ml.ensemble.BlockwiseVotingClassifier(\n",
" subestimator,\n",
" classes=[0, 1],\n",
" voting=\"soft\"\n",
")\n",
"clf.fit(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:06.761425Z",
"iopub.status.busy": "2021-07-26T20:19:06.759937Z",
"iopub.status.idle": "2021-07-26T20:19:06.909086Z",
"shell.execute_reply": "2021-07-26T20:19:06.908493Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([[0.86911216, 0.13088784],\n",
" [0.97486301, 0.02513699],\n",
" [0.99875526, 0.00124474],\n",
" [0.03272671, 0.96727329],\n",
" [0.00562858, 0.99437142]])"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"proba = clf.predict_proba(X)\n",
"proba[:5].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The stages here are similar to the `voting=\"hard\"` case. Only now instead of taking the majority vote we average the probabilities predicted by each sub-estimator."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regression\n",
"\n",
"Regression is quite similar. The primary difference is that there's no voting; predictions from estimators are always reduced by averaging."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:06.911644Z",
"iopub.status.busy": "2021-07-26T20:19:06.910785Z",
"iopub.status.idle": "2021-07-26T20:19:07.031109Z",
"shell.execute_reply": "2021-07-26T20:19:07.031478Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"
\n",
"
\n",
"
\n",
" \n",
"
Array
Chunk
\n",
" \n",
" \n",
"
Bytes
160.00 MB
16.00 MB
\n",
"
Shape
(1000000, 20)
(100000, 20)
\n",
"
Count
10 Tasks
10 Chunks
\n",
"
Type
float64
numpy.ndarray
\n",
" \n",
"
\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"
"
],
"text/plain": [
"dask.array"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X, y = dask_ml.datasets.make_regression(n_samples=1_000_000,\n",
" chunks=100_000,\n",
" n_features=20)\n",
"X"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:07.034308Z",
"iopub.status.busy": "2021-07-26T20:19:07.033258Z",
"iopub.status.idle": "2021-07-26T20:19:08.721597Z",
"shell.execute_reply": "2021-07-26T20:19:08.720190Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LinearRegression()\n",
"clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n",
" subestimator,\n",
")\n",
"clf.fit(X, y)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:08.731742Z",
"iopub.status.busy": "2021-07-26T20:19:08.731099Z",
"iopub.status.idle": "2021-07-26T20:19:08.842969Z",
"shell.execute_reply": "2021-07-26T20:19:08.843366Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([-346.86885154, -267.79712528, 107.77341861, -0.78668572,\n",
" 33.07241925])"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.predict(X)[:5].compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As usual with Dask-ML, scoring is done in parallel (and distributed on a cluster if you're connected to one)."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:08.859089Z",
"iopub.status.busy": "2021-07-26T20:19:08.852602Z",
"iopub.status.idle": "2021-07-26T20:19:10.750619Z",
"shell.execute_reply": "2021-07-26T20:19:10.749902Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"1.0"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.score(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The dangers of non-uniformly distributed data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, it must be re-emphasized that your data should be uniformly distributed across partitoins prior to using these ensemble methods. If it's not, then you're better off just sampling rows from each partition and fitting a single classifer to it. By \"uniform\" we don't mean \"from a uniform probabillity distribution\". Just that there shouldn't be a clear per-partition pattern to how the data is distributed.\n",
"\n",
"Let's demonstrate that with an example. We'll generate a dataset with a clear trend across partitions. This might represent some non-stationary time-series, though it can occur in other contexts as well (e.g. on data partitioned by geography, age, etc.)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:10.754123Z",
"iopub.status.busy": "2021-07-26T20:19:10.753451Z",
"iopub.status.idle": "2021-07-26T20:19:10.755973Z",
"shell.execute_reply": "2021-07-26T20:19:10.755584Z"
}
},
"outputs": [],
"source": [
"import dask.array as da\n",
"import dask.delayed\n",
"import sklearn.datasets"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:10.760689Z",
"iopub.status.busy": "2021-07-26T20:19:10.759432Z",
"iopub.status.idle": "2021-07-26T20:19:10.761585Z",
"shell.execute_reply": "2021-07-26T20:19:10.762052Z"
}
},
"outputs": [],
"source": [
"def clone_and_shift(X, y, i):\n",
" X = X.copy()\n",
" X += i + np.random.random(X.shape)\n",
" y += 25 * (i + np.random.random(y.shape))\n",
" return X, y"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:10.771129Z",
"iopub.status.busy": "2021-07-26T20:19:10.767125Z",
"iopub.status.idle": "2021-07-26T20:19:10.774034Z",
"shell.execute_reply": "2021-07-26T20:19:10.773636Z"
}
},
"outputs": [],
"source": [
"# Make a base dataset that we'll clone and shift\n",
"X, y = sklearn.datasets.make_regression(n_features=4, bias=2, random_state=0)\n",
"\n",
"# Clone and shift 10 times, gradually increasing X and y for each partition\n",
"Xs, ys = zip(*[dask.delayed(clone_and_shift, nout=2)(X, y, i) for i in range(10)])\n",
"Xs = [da.from_delayed(x, shape=X.shape, dtype=X.dtype) for x in Xs]\n",
"ys = [da.from_delayed(y_, shape=y.shape, dtype=y.dtype) for y_ in ys]\n",
"X2 = da.concatenate(Xs)\n",
"y2 = da.concatenate(ys)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's plot a sample of points, coloring by which partition the data came from."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:10.782894Z",
"iopub.status.busy": "2021-07-26T20:19:10.778084Z",
"iopub.status.idle": "2021-07-26T20:19:10.982815Z",
"shell.execute_reply": "2021-07-26T20:19:10.983144Z"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEWCAYAAACaBstRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAACE00lEQVR4nOyddXQc1/mwnzuzIGZZZJCZ2THFDscBO3GYGmibNIU0bUqBpphSfk3arymlTdOmYW6YHDtmZraMsiyyxbwwM/f7Y1aypN2VVriSPc85OtLembnzzmp33rkvCiklFhYWFhYWoaCEWwALCwsLi/6DpTQsLCwsLELGUhoWFhYWFiFjKQ0LCwsLi5CxlIaFhYWFRchYSsPCwsLCImQspWHR7xBCfEkIsSTccnQUIcSXhRBrevF8qUKIHCFEhO/1CiHEPb11/q4ghJgvhMhpY/tgIUStEELthnNtEkKM7+o8ZwuW0jhLEELkCiFOCiGim43dI4RY0ctydOjGKYTIFkJIIYStcUxK+bKUckHPSNg3EEL8QgjxUheneRh4Tkrp6g6ZehLf/3hE42sp5Wop5ehm23OFEJc0254npYyRUurdcPongce6YZ6zAktpnF3YgO+GW4j+Snc81fYWQggncBfQVcXTozR/GAgj7wMXCiEywi1If8BSGmcXTwA/FEIkBNoohJgrhNgshKjy/Z7bbNsKIcSvhBBrhRA1QoglQoiUYCfyrSiO+vY95jMpjQX+AczxmRYqffsuFEJsF0JUCyFOCCF+0WyqVb7flb5j5rRerXRFbiHEm0KIYt+xq5qbKYQQ/xVCPC2E+FgIUQd837daszXb53ohxI4g70GyEOJ933VtAoa32v6U73qrhRBbhRDzfeOXAz8GbvZd807f+FeEEPt913FUCPH1YO8/MAuolFLmtxof7jPHVAkh3hNCJPnm/kgIcX8r+XYJIa4JcF2Nq797hRCFQogiIcQPmm2fKYRYL4So9G37qxDC0Wy7FELcJ4Q4BBwSQjT+j3f6rvdmIcQFQoh83/4vAoOBD3zbH2y9AhVCZPre63IhxGEhxNeane8XQog3hBAv+N67vUKIGY3bfSuxrcAZvXrtNqSU1s9Z8APkApcA/wN+7Ru7B1jh+zsJqADuwFyR3Op7nezbvgI4AowCIn2vHw9yrmigGhjte50BjPf9/WVgTav9LwAmYj7ETAJOAtf4tmUDErA1279pjq7KDXwViAWcwJ+AHc22/ReoAs71yRYB7AOuaLbPO8APgrwPrwFv+N6PCUBB82sHbgeSfXL/ACgGInzbfgG81Gq+hZiKRwDnA/XAtCDnvg/4qNXYCp8ME3wyvd14DuAmYGOzfScDZYAjwNyN/5NXffNMBEqAS3zbpwOzfdeVDewHHmh2vAQ+9/3vIpuNjWj1mchv/fkNIIPN93ol8Hff/2iKT56Lm72XLuBKQAV+B2xodU1/Bv4Y7u9pf/ixVhpnHz8D7hdCpLYaXwgcklK+KKXUpJSvAgeAq5rt85yU8qCUsgHzZjiljfMYwAQhRKSUskhKuTfYjlLKFVLK3VJKQ0q5C/NmdH6I19MluaWU/5FS1kgp3Zg3l8lCiPhmx74npVzrk80FPI95s8f3lH4Z8EproYRpyroe+JmUsk5Kucd3bPPrfklKWeaT+w+Yimt067ma7f+RlPKINFkJLAHmB9k9AagJMP6ilHKPlLIO+Clwk0/W94CRQoiRvv3uAF6XUnqCyQP80ndtu4HnMBU2UsqtUsoNvuvKBf6J///zd1LKct//pEsIIQYB84CHpJQuKeUO4FnfNTSyRkr5sTR9IC9iKsXm1GC+ZxbtYCmNswzfzetDTCdpczKB463GjgNZzV4XN/u7HogBEEL8w2c2qBVC/Nh3Q7oZ+AZQ5DN9jAkmkxBilhBiuRCiRAhR5TsuqOmrG+VWhRCPCyGOCCGqMZ9maXXuE63mfgm4SggRg/l0vlpKWRRArlTMJ+3mx7eQUwjxA5+5qUqYprp42rhuIcQVQogNPhNMJeaTc7D9KzBXUK1pLY8dSPEpzTeA24UQCqYCeDGYLEHmyvTJOUoI8aHP7FcN/DaAnK3f166QCZRLKZsryfY+AxGipT8lFqjsRpnOWCylcXbyc+BrtPxSFQJDWu03GNOc0SZSym9IM5IlRkr5W9/YZ1LKSzFNUweAfzXuHmCKVzCdkYOklPGYfg/Rxv7N6bTcwG3AYkyzXTymyYNm5/Y7v5SyAFgPXIv5JBvsxloCaMCgVnKZJzD9Fw9hKp5EKWUCpiks4HUL07H9NmakT5pv/49bydqcXZgmuda0lscLlPpePw98CbgYqJdSrg8yd7C5Cn1/P435Px8ppYzD9M+0lrOj5bXb2r8QSBJCNFeSoX4GGhkL7OygTGclltI4C5FSHgZeB77TbPhjYJQQ4jYhhE0IcTMwDnNV0iGEEGlCiKuFGd7rBmqBxtDIk8DA5o5RzKe8cimlSwgxE/Nm3kgJpqlrWJDTdUXuWJ98ZUAU5hNxKLwAPIhpy38n0A4+M8j/gF8IIaKEEOMwo5man1vDvD6bEOJnQFyz7SeBbN9TP4AD03xVAmhCiCto23G7CUgQQmS1Gr9dCDFOCBGFGWb6lk9WfErCAP5A+6sMgJ/6rm088BXMz1TjtVUDtb4V5jdDmOskwf/HbW6XUp4A1gG/E0JECCEmAXcDL4dw3kaFPB3Tz2LRDpbSOHt5DNOJCYCUsgxYhOmQLcO8KS6SUpYGPrxNFN88hUA5pj37W75tXwB7gWIhROPc3wIeE0LUYPpc3mgmVz3wG2CtLxpndvMTdVHuFzDNGAWYDu4NIV7fO5irm3d8prhgfBvTFFaM6VR/rtm2z4BPgIM+GVy0NNm86ftdJoTY5jO9fAfzvanAVKzvBzuxzxfxX3z+l2a86BsvxnQaf6fV9hcwlWEooborgcPAMuBJKWVjwuUPffLVYK4wXw98eAt+ATzv+x/fFGD774Cf+Lb/MMD2WzFXioWY/5+fSylDVQJXYwaEFLa7pwVCSqsJk4VFRxFCHAG+LqVcGm5ZguELdlgNTA3V4SyEuBO4V0o5r419soFjgF1KqXWHrOFECLERuNvn77Noh76QWGNh0a8QQlyPaWP/ItyytIWUsgQIGoDQGp/J6luYoatnDVLKWeGWoT9hmacsLDqAMMuuPA3cJ6U0wixOtyGEuAzTX3KSACHEFhaNWOYpCwsLC4uQsVYaFhYWFhYhc8b7NFJSUmR2dna4xbCwsLDoV2zdurVUStm6csSZrzSys7PZsmVLuMWwsLCw6FcIIVpXWgAs85SFhYWFRQewlIaFhYWFRchYSsPCwsLCImQspWFhYWFhETKW0rCwsLA4w6isrKSoqAiPp612KJ0j7NFTvgYwW4ACKeUiX2Ob1zGLj+UCN0kpK3z7PoJZvVIHviOl/CwsQltYWFj0QRoaGvj0008pLy9HURQMw2DGjBlMnty651Tn6Qsrje9itoNs5GFgmZRyJGb1zIcBfKWlbwHGA5cDf/cpHAsLCwsLYMmSJZSWlqLrOl6vF13X2bp1K3l5ed12jrAqDSHEQMx2nc82G17M6baYzwPXNBt/TUrpllIewyzJPLOXRLWwsLDo09TU1FBaWkrr0lCaprF79+5uO0+4Vxp/wux/0LzwW1pj+0zf7wG+8Sxa9hvIp2XnOQsLC4uzFpfLhaIEvqU3NHS5FXsTYVMaQohFwCkp5dZQDwkwFrDaohDiXiHEFiHElpKSkk7LaGFhYdFfSExMDDiuKAqDBw8OuK0zhNMRfi5wtRDiSswOYnFCiJeAk0KIDCllkRAiAzjl2z+flj2JB3K6J3ELpJTPAM8AzJgxwyrja2Fh0WeoqKhg27ZtlJaWkpCQwNSpUxkwYED7B7aDzWZjzpw5rFu3Dk0ze2OpqorT6WTSpEldnr+RPlEaXQhxAfBDX/TUE0CZlPJxIcTDQJKU8kFfH+JXMP0YmZhO8pGN/Y2DMWPGDGnVnrKwsOgLlJaW8v7776PrepPvwWazsWDBAgYOHNgt5yguLmbXrl3U1dUxaNAgJkyYQERERIfnEUJslVLOaD0e9pDbADwOvCGEuBvIA24EkFLuFUK8gdnLWcNsgtOmwrCwsDh78Xq91NfXEx0djc3WN25169evb1oFNKJpGmvWrOGWW27plnOkp6eTnp7eLXMFok+8k1LKFcAK399lwMVB9vsN8JteE8zCwqLfYRgG69ev58CBAwhhukInT57MtGnTml6Hi2A+1pqaGjRN6zPKrS3CHT1lYWFh0a1s2bKFnJwcdF1H0zQ0TWPnzp0cOHAg3KLhdDoDjquqGjTyKRRcLhdlZWU9kgHemr6v1iwsLCxCRErJnj17ApqAtm/fztixY8MkmcmkSZPYvHlzC/lUVWXs2LGdUhq6rrN69WqOHDnSlAE+YcIEZs6c2WOrKmulYWFhccag6zq6HtjV6XK5elkafyZMmMC4ceNQVRW73Y6qqgwfPpxZs2Z1ar6NGzdy5MiRFhnge/fuZe/evd0s+WmslYaFhcUZg6qqREdHU1tb67ctJSUlDBK1RAjB7NmzmTZtGtXV1cTExHQqsglM382BAwf8lKSmaezatYsJEyZ0h8h+WCsNCwuLMwYhBOeeey6q2rIsnc1mY/bs2WGSyh+Hw0FKSkqnFQaEb1VlKQ0LC4sziiFDhnDllVeSmZlJdHQ0Q4YM4eqrr+6WBLq+hM1mIyYmJuC2nrxWyzxlYWFxxpGRkcGiRYvCLUaPIoRg3rx5fP75502OdSEEqqr26KrKUhoWFhYWPqRhYOSUYeRVQpwT26R0RLSj0/O53W527NhBbm4uDoeDCRMmMGLEiG6LbBo0aBCLFi1i+/btVFZWkpKSwvTp00lISOiW+QPRJ8qI9CRWGRELizMXj8dDWVkZ0dHRxMXFdWku6dXxPLcdWdYAXh1sAhQFx5cmoQyK75Rsb7/9NnV1dRiGWcjbZrMxZswY5s6d2yVZe4P+VEbEwsLCol22bt3Kjh07mvITUlNTWbBgQaedy9qGE8jSetB8nRo0Ceh43t6H87uzO7w6yMnJoaGhoUlhgBnZtH//fiZPnkx0dHSn5Aw3liPcwsKi33H06FF27tzZIj/h1KlTfPHFF52e09h96rTCaE6DF1ne8X4UBQUFfkmGYJYqP3XqVIAj+gfWSsPCwqLfsWvXLr8bsmEYFBYWUl9fT1RUVMcnVYOsJCSgdNwHERMTgxDCr5OelJKoqCiOHz/O4cOHURSFUaNGkZXVP3rKWUrDwsKiT+OqLmfHi0+Qv3kpKArZ8xbREDE84L6KouDxeDqlNNRpGWhLj4K35WpDJEagJEZ2eL7x48dz8OBBP+XmdDrZuXMn+fn5TduOHTvGuHHj+lQuSTAs85SFhUWfRfd6WPLjm8hd+yHehlq8ddUcWfYWsmBvwFpNiqJ02iGuTs9EGZYIdgVsCjhUiLJjv3F8p+ZLTEzk4osvJiIiokWyocvlIjc3t4Uy0TSNvXv3UlVV1alz9SbWSsPCwqLPcmLjEtw1FUj99A3W0DyoOauwzR2GhtIiMmnevHmdrhYrFAXHzRMximowTlQhYp0oo5IRauefrYcMGcLtt9/O6tWrOXz4cJtZ3AAnTpwgPr7jkVq9iaU0LCws+izlx/aiuer9NzRUMy1FwZU+noKCAmJiYpg4cSKp9U60zQWIlCiU7IQORzyVlJSwe/9uampqyFKymOCNJULtfKkPMBPujh492qayaNzPbrd36Vy9gaU0LCwsOoRRXIP2xTGMwhpEnBPb+dmoo3umGGB85jBUZyS6u2X0kmK3k5yVTdb0mQDIBi/u57dTVFOJG41UPQpnYjSOu6YgIkK7ER89epQVK1Y0mY1KS0vZv38/119/fecc6z6klHi93pD2HTp0aKfP01tYPg0LC4uQMYpr8Ty3HeNwOdR7kcW1eP+3D21rQY+cb/C5V2JzRIA4fasSioozNpGMqec1jZV9tJv/GXtZEnucVbGFvJl4iH21BXg/OxzSeQzDYPXq1S38DLqu43a72b59e5euQVEUkpKS2tynsU+4w9H57PPewlIaFhYWIaN94R9dhNdAW3YMaQTIcegi9ohoLv31a6SOmY5QVISqkj75XC597GUUxXQuG4bBkpKd1KpeNEXiVQx0IdkWXUJxzvGQzlNdXR3QfGQYBnl5eV2+jkCVdxtRFIUJEyYwcODALp+nN7DMUxYWFu1wBPgU0DAKpwTeRTOg1gNxXbP/ByI2fTCX/OIFdK8HBKi2lk/jJSUluIQGrdwXupAciKhgcAjncDqdLTK3W2/rKhkZGcyYMYNNmzb55W0YhkFFRUWXz9FbWCsNCwuLNngJuBN4AXgJEZ8ffNfInnXiqnaHn8IAs8aTCBQxJcAdHdotLjIykoyMDL/IK5vNxqRJkzolb2sGDRoUMLJLVVVSU1O75Ry9gaU0LCwsglAM/B1wAzogsZ33Gdg9LXezKahT0hH2wOaXniYtLQ0jwJ1MlYKhk0eHPM/FF19MSkoKqqricDhQVZWJEyeSlZXFtm3b+OSTT1i/fj3V1dWdkjMpKYmMjAw/M1Vjj/D+glXl1sLiLEDzuDixcQn1pUUkDZtA+sQ5gZ/OW/AW8CegZRc4bdsstGXXgtd86lenpmNbMKJL+QxdZffu3WzetBnNl8+hCoW4iGgWL1iII61jyX6VlZXU1dWRnJyM1+vlf//7H5qmoes6iqKgKAoLFy4kLS2tw3JqmsbmzZvJyclB0zSysrKYO3dun8zNCFblNmxKQwgRAawCnJi+lbeklD8XQiQBrwPZQC5wk5SywnfMI8DdmI8935FSftbeeSylYXG2U114jKU/+xK6143mdmFzRhCXNZyLf/ZfbBFthZK+C/wBaF2sT0UaX4bar0KkrUdXGIauUbxrHQ2VJaSOmkpc1rCg+xYXF7Nn+07qc0sZ7IphhDsem6GgjE3Ffs2YTvWwWLZsGUeOHPEbT0xM5MYbb+zwfP2Jvlga3Q1cJKWsFULYgTVCiE+A64BlUsrHhRAPAw8DDwkhxgG3AOOBTGCpEGKUlLLtjBkLi7OcdX/5Ee7aSvA9IGqueirzctj37r+YdMt32zjyAuCJAOM2hHI5xHXMQSylROZXI70GyqC4dpVNTfFxlv3iTrwNdSANpDQYNGsBs7/1eMBVUnp6Ookn8pGlTrPIIAAGxoES9O3x2KZldkheMDO0A1FVVYXH4+kXIbLdTdjWk9Kk1vfS7vuRwGLged/488A1vr8XA69JKd1SymPAYWBm70lsYdH/cFdXUJV3sElhNGJ4PRxb9V47RycAj2EaAyKBCMABfBvoWBKacbIW95/W43l5F9439+B+ci3aruI2j1n95P00VJaguerQ3A3oHjcnNi3l6Mp3Au4vq1xmP4zWxhOvgb6lsEPyNtJWhnZny5X0d8J61UIIVQixAzgFfC6l3AikSSmLAHy/GzukZwHN1X6+byzQvPcKIbYIIbaUlJT0mPwWFv2bUEzTFwMfAT8Cvodpsrq1Y2fRDTwv7oQaD3h0cOtmbseHBzFK6gIeU3vyBLUnT/gpO93dwOElrwY+j9fwC7ttwts5g8S4ceP8HNeKopCdnY3NdnZmLIRVaUgpdSnlFGAgMFMIMaGN3QN9HAJ+6qWUz0gpZ0gpZ/SnUDYLi+7GGZdI/KCRtP76KHYH2fOvDnGWBOBq4HpOP8OFjnG0InBzI91A3xZ4BaB73QgR+PakeVwBx0VyJDgD3MhVgTKu43IDTJ48mezsbFRVxW63Y7PZSE1N5bzzzmv/4DOUPqEqpZSVQogVwOXASSFEhpSySAiRgbkKAXNlMajZYQOBzq05LSzOIube/wSf/+xLGF4PmrseW0QUselDGHftvb0jgMu/ex0AEmRd4JpMcZnDsEVGoblbFitU7E6GzF0Y8BghBPZrx+J9bTcYEnQJdgURH4Ft7qCAx7SHoihcfPHFVFdXU15eTmxsLMnJyZ2a60whnNFTqYDXpzAigSXA/wHnA2XNHOFJUsoHhRDjgVcw/RiZwDJgZHuOcCt6yqLvUwloQCeK/lUeh81PQ2kODJkH0+6BCP/wTc1VT96Gz6grKSR5xATSJ89rKsPR08hqN+6/bAS91WrDrmC/egzq+MCrgOI9G1j1+28hdQ1D86I6o4hJG8ilv3oFe0Tw/tqyyoW2rQhZ5UIdmogyPhVhC08OSX+mL4bcTsJ0dKuYZrI3pJSPCSGSgTeAwUAecKOUstx3zKPAVzG/YQ9IKT9p7zyW0rDouxQDPwH2YpqPMoFfASEmeuWtgxcvA8MDugdsUabC+PpWiM3oKaE7hfeLo+gb80/XrbIriLQYswptG/kddaVFHF3+NnWlhaRPnMug2QsCZoV3FMMwEEJ0Kgz3bKHPKY3ewlIaFn0THTMg8BTQ/Ak8CtPZ3HZVVKSEv46BsoMtxxUbTL4LFj/bjbJ2D3pOCd5PD0OV2xzIjsexcDRKcufLjneU0tJSVq9eTUlJCaqqMmrUKObMmXPWOrXbIpjSODtjxiwsws4GoJqWCgNMZfJB+4fXl0Jlrv+4oUHO+12WrifQ1p0wixr6kLmV1P55Bf+7fRZf/OqrVBce69Hz19bW8sEHH9AYUanrOgcPHmTJkiU9et4zDUtpWFiEhWL8FQaYOa8h9KawtVFN1hHc3h8YD/5Z392LUViDLK41ndM+BAJV2BgcN5uTezaw5NGbaags7TEZdu/e7Vf+XNd1ioqKqKys7LHznmlYSsPCIiyMDzIeCUxu/3BnLAy7FJRWyWf2KDjnWyHKUImZf3EeZvb3ncChEI/tGLKsHgL4D1TFQULkIKTNgUeN4NDnr/fI+QHKy8sDlj9XVZWqqqoeO2971NbWUlRURENDzyru7sIy5FlYhIUxwDRgK+bqAsyiCKnAJaFNcc1/4cVLoeyQ2dnO8MKoRTD7eyEcLIFv4q4+Qs6nLk7u0YlO3caYhXeRNOwDoHvDSkVKlF+iHkCDdLE1DWpH3guGweYyQfSRIwwfPrxbzw+QmppKUVGRn+LQdZ3ExMRuP197aJrG0qVLKSgoQFVVdF1n9OjRnHvuuX3aQW8pDQuLXqZo7ynWP7eVyvxzmHqDk/FX7sUeKRHiUuAezLIdIRCdAl/fBoVbTP9G+lRIHhGiFJtoqMjlk4dq8Nab+qb0IORvqmDOt59g0KzHO3dxQVAyYhEZsciC6iYTlSENViWepMoRZTrwFdOjs3LlSmJiYjpVRbYtxo8fz759+/B4TvtVVFVl0KBBxMV1rBJud7B27VoKCgrQdb3JbHbw4EHi4+OZOHFir8sTKlb0lIVFL3Iyp5QPf/o5mvu0bd3mVJl28ySmXh/MZNWdaMD/A95m87/rOLJMo3WmkzPWwTXPbAs5j6PsyB72vfsM1YXHSBk5iXHX3Ets+hC//aRHR/v8MPquk0ivxnF3DqsGSlD9n12zs7NZsGBBJ66vbSoqKli/fj2FhYXY7XbGjh3L9OnTg7Zi7Sl0Xee5554LaC6LiYnhtttu61V5AtEXq9xaWJx1bH55RwuFAaC5dba/uZtJV49B7fFGRn8B3gM0irbrfgoDQPNI6k6eIDYju93ZCrevYs0fv4vudYOU1BQeI2/9Zyz49au+8iWnEQ4V+8LR2BeORnM3UPziRvDEEOg2VFNT06mra4/ExESuvPLKHpm7IwTqR96I2+0Ouq0vYDnCLSx6kfLcyoDj0pDUVwauqdR9eIG3aWyq5IgJbDeXhsAe3b65RkrJln8/hu5xNfkrpKGjuerZ/vKTbR5rc0Yy984foDr9czQURSEzs+NlzPsTdrudmJiYgNv6+rVbSsPCoheJz4wNui0yvo0w2m6hDtNrYDJmkR21lftEqDYGjD2HiLh2kgsBb30N9eUnA2yRlB7Y1u7xDoeDKVOmtEisE0Jgt9u7rS93X0UIwfz587HZbE1Ob0VRcDgczJo1K8zStY1lnrKw6EVm3DaZT3+1HM3T0qcxYeEYbI6eNk3FAbFAOQBDzlWpzLOR87GGalcxNDuJ2WOY+51AjZf8UZ0RKIqKrvsXJHTEJoQ0x7Rp04iPj2fXrl00NDQwcOBApk+fTnR0R3NN+h9ZWVlcc8017Ny5k4qKCtLS0pg8eXLQFUhfwXKEW1j0Msc25LHu31upK6nHFmlj0tVjmH7zJIQSSphlHrADs8zIbDr+3Pcp8GtO9/0WuGvsVBx/gKikc4jL7FhzpU3/+jm5K98zfRo+VGckU29/kJELbumgbBZ9CcsRbmHRSE0R5HxgJpuNvhpiuje0sz2Gzh7M0NmD0b06ik0JMSZfAr8FPsa0KgvMTnr/pGNd9C7H7I/xDGZngbE4Y79O+oQxHbmEJqbf9WO8dTXkb1mGanNgaF5GXfYlRlx6c6fms+j7WCsNi7OLrf+CT75jJsOB6cBd9DRMuSu8crXLp8BvaFnuQ2C2lXkd2AhUYSYM9r4jtaGylPqyYmIzhuCICu63seg/WCsNC4uKXFNhaK2ilD78Bgy9GOIHhkWs0Hgb//pQErNK7uWY+RcSpA6lE+CD/eZ1TvwSzPg62CNPH1XvQd9XAm4dZVgiSkbXb/KRCSlEJnSiH8gZgpSS/Px8cnJyMAyDESNGMHTo0D6d2d1ZLKVhcfaw700IkEwFwP63YfZ3e1eeDhGsLpGH02VIMBcf8Vsg8ijkVMGpvbDnVfjqWlBt6EfK8b6xx9Q3hgErFZTxqdivHnNG3uB6i/Xr13PgwAE0zQwKyM/P58iRI1xyySVn3Ptqhdxa9Eu8DV68DYFbhQZF9xKosqw0DPTWq4+ewOuCJT+CxxPhVxHw8kIoOxziwZcTuLxIAPOyQ4EZqebfWgOc2gcH3kVqBt639pqNkDTDfCs0A2NfCcbBss5dkwUVFRXs37+/SWGAWVfqxIkTFBUVhVGynsFSGhb9iuqTtbz3yGc896U3+O+X3uC9Rz6jujjE7OExi/2rwgIeqfHdws/5y7Y/4e6K8tA8sOHP8I+p5s/Gv/oUlY/Xr4NNfwVXJehuOPwp/Gsm1JWEMPkNwDDMKrhgGgkcmM7wADiafbW9tXBkCcbxysD7eg30ncUhyGARiIKCwKXsGxVHR5BS0tf9zJZ5yqLfoHl03n3wU1zVbqQhkcDJA6W8+9Bn3PbMNdic7XycB4yHuT+AdX9A6m4MaaAJhTfThpPncFKcv5JqTzWPzv5Zx4WTEl66Ago2gLfeHFv6kNkQ6Y7PoPQA5K5o6U+RhrkS2PJPOP8n7ZwgAngO+AJYDwwArsQsZ94Kjw67y0+/Vh0Ql9WO/O2c3iIoDocjoAmqMVkvFGpqalizZg35+fkIIRg6dCjnnnsuERE9nfDZcSylYdFvOLY+D82lIY3TdzhpSDSXxrH1Jxh5QQihpxf9CsZex+ZlP+B49QlWJ6aTG2mWzPAYHrad2kZJfQmpUakdEy53BRRsPK0wwPz7xDo4vspcTah2U0k0R3NBwaYQT2IDFvh+Gvk58FPMTG8NPAacbIDtzcxNig2mfAUlJj6wcrArqJN7N+y4p5FScuDAAfbs2YPb7WbQoEFMnz69RxLnsrOzWb16td+4EIKRI0cGOKIlXq+Xd999F5fL1bTSOHbsGOXl5dxwww19zidiKQ2LfkPNyVq8bv/sY69Lo+ZUbegTZUzltYETOVwZ6bfJrtgpaTjVcaWRt6alwmhEa4C8tbgyL8Wpa/h9/VUnpAVuulSVf4Q9//s7ZYd2EZs+mPHXfoMB485ptddFwAjMvuLl0DAE3v0NKFGgKuYq4/qXIWEwArBfP870a0hMv4ZdQRmTgjL6zIp8au2YPnjwIMePH+fGG28kMtL//94VDh8+7GdSEkJw4YUXhqSkDh8+jNfrbTGHYRjU1tZSUFDAwIF9K6rPUhoW/YbkYYnYI2x4G1oqDnuEjeShHWuiMzpxLLlVuWiy2VyGIKIkitjKeGSS7NgTXky62TXPW9di2FAj2PJeMTtzD7Fo6GAGRB9BFc38HDYnnPNNv+kqjuew9Ge3obtdSGlQdyqfkgPbmPPt3zNo1qWt9h4MfMf8Mx749pfNqCnNBelTWpQeV0cmo9w/G33fKXBpKMOTULJ6v5dET9LQ0MD+/ftbVJKVUuLxeNizZw/nnNNa8bZE13VOnjyJEIK0tDQUJbjrt7a2lvXr1/uVOFcUJeTGTuXl5S2c6I0YhkFlZaWlNCzObspd5Sw9voSiuiLGJ09g/sDzcLaumheEQVMziR0QQ2VhNYbX/JIqNoWYAdEMmtaxhLbrRl7H8hPL0DUdiSSlIJ2pq+biMBws+WQ10clRXPbjC0gcFB/ahONvgiU/9Bv2umHP0YkYhsHHR37AuQNfYmTSOhRFRwycDYv+AXH+su985Q9orpYrF93jYutzv2bgzHbCOIWAtAnBN8c4sM3sWzciMKPY5Kl6c/WT7F/9NlTKyspQFMWv/LhhGBQWFrZ5bF5eHsuWLWt6rSgKl112Genp6QH3z83NDThuGAZHjx5l+vTp7cqbnJyMzWbzUxwdUTy9Sdiip4QQg4QQy4UQ+4UQe4UQ3/WNJwkhPhdCHPL9Tmx2zCNCiMNCiBwhxGXhkt2ic+SUH+Abn3+N13NeY1neUp7Z9Q/u/+I+ajyhRT8pqsLixxcw/vJRRMQ7iYh3Mu7ykSx+/DIU1fwoL993knv/vZGb/7KGv3yWQ0WdJ+BcqVEDePL8PzIj/RwSXUnM+OI8HC4neASaW6eqqIb3H12CrgXJ62hNRDzcuQzih4A9GuzRuO2ZfHzsx3gN0xyiGRGszLuH5w88T9G1eXD3WkgL3KGt9NDOgOPumgo8tZWhydSP0A+V4f7jejz/3Y7nn1twP70Jo7yl0pRS+plxAhETExPwyR3a7lVRV1fH0qVL8Xq9TT9ut5tPPvmkRbe/7mb48OF+znRFUYiNje2TZdLDudLQgB9IKbcJIWKBrUKIz4EvA8uklI8LIR4GHgYeEkKMA24BxmPWSVgqhBglZaA2MhZ9DSkl/2/rH3Dpp6OHXLqL0voSXs95lXsm3hvSPI4oB3PvmcHce/yqG/Ds8sO8tDYXl9f8SBRW1LNkdxEv33cucZH+obYDYwfx09k/Z0vuLrazB6N5DocE3aNzYlsh2aE+lWfNgAeOmZFSCNa8UMKpmuMBdhTUFZfB8V/D7pdB98DIK+DypyBhMAARcUl466r9j1QUbBE9VwFWSkldXR02m63XIneM8obT+SONcpTW43l+B87vzgEBW7ZsYefOnRiGgRCC4cOHc+GFFwZcccXHxwdVLLW1wX1fhw8fDthJT0pJbm4uo0aN8tuWnZ3Nxo0b/cYVRWHo0NBqgtntdq655hrWrVtHXl4eiqIwfPhwZs+e3eec4BBGpSGlLAKKfH/XCCH2A1nAYuAC327PAyuAh3zjr0kp3cAxIcRhYCZm/KFFH6fcVU5Jg38+giY11hasCVlpBKOq3sMLa47habYy8OqSqgYvb2/K4yvnDw96bH1ZPUaAFYU0JA2VwTKxgyAEpI4FIH2cIHdjvl+nPsMwGHrwK1Cx18zXADM098Q6+M5hcMYy7pp72fKfX6G7T59fdUQw7IJrUe2hhXF2lPz8fFauXNkUxZOZmclFF13U48pD31bY1De8CQm4dYxjFeyqPs727dtPb5KSw4cP09DQwMKFC/3ma2sl0tY2l8sVUGkYhhF0hRITE8Ps2bPZsGFD09xCCKZOnUpSUvs9SZrP0xPtbXuCPpHcJ4TIBqZiVl1L8ymURsUywLdbFtA8UybfNxZovnuFEFuEEFtKSkJJnLLoaeyqPWgqgEPt+k3wYHENDrXlxzmx3svcQxU0PLWef9/8Gsv+uJaGAN3xsqZkYIvwf36SEtLHDfAbD5VRFwwjItbZZDoDUB0qEyaVY6vKOa0wwMzZ8NTCzhcBGHr+NYy75l5UZyS2iGgUu4PBcy5n6l0Pd1qetqisrGTJkiXU1dWh63qT/f/jjz/ukfM1R1a7wQj86ZC1brZtC9zQqaCggPp6/4g1RVHIyMgIeMygQYOCyjFw4MAWDaEaEUKQlRU8z2X8+PHceOONnHPOOcyYMYPrr7+eadOmBd2/vxN2R7gQIgazGtsDUsrqNpZjgTYE/KRJKZ/BrP3MjBkzrLSlPkCcI45RCSM5UHEAQ55+mnOqTi7LvqLL86fEONF8T4mxbp2LjlWR6DKf8AWg6RpH1uRycv8pbv771S16cWfPGkRC1l4qTlSh+5oj2Zwqw+YOIXFgiI7wANgj7Vz3hyvZ/MpOcjecQHWojL1sJFOGrIXPA3wsvfVQZN4ghRBMuO4bjFl4F3UlBUQmpOKI6bws7bFnz56AjuPKykpKS0tJSem5kFx1eBJGTmkL85QpgISBcegbg1ugy8rKiIryd5rPnz+f9957D03T0HUdm82G3W5n7ty5QefKzMwkMzOTwsLCJp+IzWZj+PDhREVFsW3bNgoKCoiNjWXixIkkJyc3HRsXF8fkyYFDp880wqo0hBB2TIXxspTyf77hk0KIDCllkRAiA7OMJ5gri+aPCQMxGwJY9BN+eM5DPLL6Iao9VU2KY3LqFK4evrjLcw8dEEN2agyHi6q54nAlkV7D7ylD6hJXtZtj608w4rzspnHVprD4dwvY+8lBDq88huqwMe7ykYw8v2MNiQIRmRDBed+axXnfatbCM7f8dGn25tij/HI2bM5I4geO6LIc7VFVVRXQdCOEoK6urkeVhjI+FbH+BLK8HjSJgaQoqgHvkBgGOmXAyKJGKioqAq4eEhISuOWWW8jJyaG8vJzU1FRGjhzZZoa2EIIFCxZw9OhRDh48iKIojB49mgEDBvDWW2/hdrvRdZ3i4mKOHDnC/PnzGT58OKra0x0X+xZh66chzCXF80C5lPKBZuNPAGXNHOFJUsoHhRDjgVcw/RiZwDJgZHuOcKufRt9Clzq7SnZR2lDCiISRDI3v+o25kfJaN7/7y1oGbyrCEcTcATDl+vHMvGMKe8v2cLDiICmRKczOmNMtZrKQkNKsTVWyHwxfVI5QICLR9GlEJvSOHM3Yvn0727Zt81ttqKrKLbfc0uPtV6VHQ9tUQNme4yxRDqMrIBXTB5GamkpxceDaWBEREdx+++1t5lJ0lbVr17J///6A/g5VVRk7diyzZ8/uURnCQV/sp3EucAewWwixwzf2Y+Bx4A0hxN2YvS1vBJBS7hVCvAHsw4y8us+KnOp/qEJl6oCpPTJ3UoyTu88ZwprtJ9HdgT8aqlMhJjOaH695mCOVh/EaXuyKg2d2/YP/O+9JsmLaqdHUJU4Cm0FEwZc/hY9/6CvXrkP2BWbORhgUBsC4cePYu3cvDQ0NTSsOm83GyJEju6QwvPW15HzyIvmbl+KIiWf0FXeQNf1Cv/2Ew4bt3MEsO74GV51mVuD13aNLS0tRVdVPoYFZFLC2tpa4uOAJilI3kDVuRJQD0Yk+7Hl5eQEVBpiJgI2JhPPnz+/w3P2RcEZPrSGwnwLg4iDH/AazfZmFRUDSxqQELb5nYOBWXKyJX8qhooN4fE/5ut6AW3fxxObH+dOFf+khyf6FWXBQBRSIFHD9U3Ddi+bKI8xPqU6nk+uuu45t27Zx/Phx7HY7EydOZMyYzrWBBfC66vj0kRtoKCtu6iFeenAHY6/6ChNvvN9v/9LS0oBRSpqmYbfbAyoNwzDaNDlpG/PRVhwz/SMS1Mnp2C4fgVBDf7+dTic1NcFziXRd5+DBg8yaNSvkAoX9mbA7wi3Cz54Tlbyw5igF5Q1MHpLInfOGkp7QvfV5eovEgfEMnTOI3A0nmkJdpU+LlGacZM+8zRjFWpPCaEQiOVFzgnJXOUkRoYdKhsZ2TEts6wSx74H4DIR/Dkk4iIqKYt68ecybN69b5jv6xf9oKD+JJkFPHITw1CPrytj33r8ZddntOONOZzs35kIEUgyNstXV1bXwbSiKwsCBA4OGBOt7T6F9cbSFg13fVQyqwH55+4UEG5k4cSKrV68O6lcB0x/icrkspWFx5rN8bzG/fGc3bq+BBI6X1rFkVyF/iz9OwjuvgaISfcvNRN9xO8Ie2s1NN3Q0w4vTFp6yzhc9cC6bP97Oyv+tRWgKhdl5HBufg+40az4JPXjClJQhZoB3iHdp0V2vCR3YCszugXOGn8LtK6jPmIBn6BwwdISi4nR7yDi6j9KDO8macUHTvqtWreLIkSMBnfE2m43x48fjcrnYuXMniqJgGAZpaWlcdNFFQc+vrT7uH5HlNdC3FWG7ZDjCFtpqY8SIEZSVlbF3714MwwgaMNDTfp++gqU0zmIMQ/LER/txNftiaYakvsHLvw4U8b39BwCo/u3vcC1fQfIL/20zQ9WluXhm1z9Ymb8CXeoMjBnIfVPuZ2zyuG6Rt9pTzbGqY6REprTpexCKYPLl4/g/41d4Df/yD4kRidR4avAaLTv/pUenkxzZE1FCLoI3rOiFjoFhwpuUjScpySyYqNrMfL1IG6Wj5jB7mcDIrkNJiaayspLDhw8HXGU01l8aO3YsqqoyadIkysvLiY6OJja27d7msiZ4yRDcGthCWxUIIZg9ezaTJ0/m2LFjrF+/voWsNpuNadOmnTVRVGeWu9+iQ5TWuKkLUGrcUBT2pJ0umSAbGvBs2IBna+Akq0Z+t+k3rMxfgdfwYkiDvJo8frbuJxTWBu5sFipSSp7f+xxf+fROfrfx13x3+bd5aNWP2qxZ5bRFcMmQS3G0KoboVJ3cM/FeBscOJkKNaBqLskXzwxkPdUnO4FzK6Y57zdEA/3IoZwr1qaPNHiLNkAKqbRo1mhvvG3uRUlJcXBz0YSQtLY3Fixc33ZAdDgfp6entKgwAkRFkH4cKUR03CUZGRjJu3DgWL15MVlYWDoeDhIQE5s+ff9bkaIC10jiriY6wBUvEJc7V8oYsPR48WzbjnBG4amdhbSF7S/f4Pb1rhsb7R97jG5O/1WL86KlaXl2XS25pHVMGJ3LznCGkxAaudrsyfwUfHv0Ar+Ftmv9QxUH+sOUJfjH3saDX97WJ96IAnx//HACH6uTL47/CvKz5zMmcy9biLeRU5JAalcr8rPOItveUeeFC4ANgG9CA6Qy3AQ8C3d8UqCNUVFSQk5ODx+MhOzubQYMGdVu9I10EfvJWELgVHVnlQpY3EBkZGbTzXUZGRqdDWe2XDMPz3PaWJiq7gu3S4V26xpSUlIDlS84WLKVxFhPttHH+2AGs2n8Kj94sS9vr5ppdn7bYVzidqKnBS2qcrC/Gptj8HMy61Mmtzm0xtulIGQ++ug2PZmBIOFBYxbtbT/Dfr88hK8k/u/fdw//Drbc0NWhSY3fpLqrdVcQ5A2dK2xQbX5/8Lb484W5qPTUkRCSi+m5kqlCZmTGLmRmzAh7bvajA/wPWAcuBOOAqzJ7f4ePAgQOsXbvGZ6eHw4cPkZmZxYIFC7ol52DIkCFUVlb6mZ0kkiSvE+yALhk0aBA2mw2vt+UDR2NyXWdR0mNxfHUa2vJjGIU1iIQIbOcNQR2R3P7BFkGxlMZZzo+vHo/Lq7PpSBl2VaBpBot3f8F5hzc07XMsaRCbR80iKWkCl5TXB7yxD4od7LfKAPPGPSbxdNimlJLfvb+3hR/Fq0t0Q+PppYf49U3+y/xgZihFKNRp9UGVRiNO1YkzMrSeHT2HAszz/XQOqRvg1cFp6/JqwOPxsHbtKprfzzVNp7DwBLm5uQwb1nWFNnHiRA4eyKGhvgEdwwx5lYJzqgegopjXkRqFEIJFixbx2WefUV9fjxACRVG46KKLQjJDtYWSFoPjlsDl5y06h6U0znKinDaevG0aJdUuTlW7yE6Jwb4vlfJvrMeoqOS/Uxbz6ejz0FQ7yqpc/rPmON+7YgzXzGhZuiElMoV5WfNZV7i2aVUgEDgUJ1cNv7ppv6p6L6U1/s5fQ8Kmo2V+4wDT0maw9PgS9Fa5nBG2SAZE+VY/uhf2vA7734bIJJjxDchqu0Nbf0F6dbyfHMLYc8p8o+Kd2BeOQh3W+dDgwsJtKIoHXW/pDNY0yZEj+7qsNKRLgzf3syg/k5zISgpsNUQZdsbWJZAqo8EOjuvHNSm/xMREbr75ZioqKtB1neTk5F7NsNZ1nV27djUl6g0dOpQZM2b0Wnn4/oSlNCwASI2LIDXO9wWZNpW0jRvYtXEvS5YU4vGVrTYMiWZI/vjJAc4bM4CkmJZP79+Z+gADYwby0bEPqffWMyl1Ml+ZcHeLiKQIe/AIk9gAlWYBbhl9KxsK11Gv1eM1vKYyUh3cN/nbprlJ88DzF0LxTrPdqlBgz2tw6e9h5n1dfGfCj/ftfRhHK8ye3gAVLryv70F8dRpKWmg+Ec1Vj7umgsikNBTVhqruDBLQZWC3l7Y7n/TqGPtLkJUuRHosyogkhHJ69eN9Zx8yrwqHLpjoTWQiiaAKxJAElOwEbJPTEa18WEKIDpUT704+//xzCgoKmkxpBw4cIC8vj5tuuilg5duzGevdsAiIEIIVVTbcrfscAIoQrD1YwlXTBqLpBq+sy+W9rfl4dcnF4yfzl/OvIzZA0yOACIfK+WPTWLn/JN5mc0fYFW6ZMyTgMcmRyfz14qf54Mj77CrdSVpUOteMuIbhCb5CfnteO60wwCwz7q03269O/FLYSnN0B7La1VJhNKIZaGvzcFzXdjizrnnY+txvOLbyPYQiUFQ7k2/9HkPPB+F1gmj5/1WlwqhRbYeiGuUNeP6zzZTJo4NDRSRG4PjyVITThqz3mDK3/uzoElwa9nmB/8/hoqysjMLCwha+F8MwcLlcHD58uEtZ8WciltLoBtwbN1L38qvIujoir15E5MKFiH70dCINA4Tws5MrCIQwq1w0R0DTvg++up2tx8px+25qb23KY83BEl7+1rk4giRPPXL1eKrqPezKq8SmKnh1gyunZHH9OYODyhjvjOf2cXdglitrxb63TiuM5qgOyFsNo68KOm9fR1a6QBVmdG6LDSDL/HtJtGbbf39L7qr3MXxlPHRcbH/p9yTUP8pFNXaWxZQ0ZcwbAiY0xJLW0LZZz/vufmjwnl6peHRkaT3aylzsC0YgGzRQhL/SAGSQ9rvhpLQ08MpK0zSKioospdGK/nNn66NU/+kpav/6N6TLBVLiXrWK+tdeJ/mlFxF9PNlHy8uj8qFHcK9ZA6pK5JVXEP/rX6MmmeUdFkzK4M1Nx/FoLb/8hpTMH53KgcJqtuWeVhhgOrVLa9ws21vMFZMD9zeOdtr4y13nkF9eT3FlA0MHxJAc0wVHdWQipipr3f1NgqNrjtRwI1KiAt58UUDJCl6kD0BzN3Bs5bvonpaRZ7rbRfmmowyJmMmN9bEUOOvwKgYZ7iiihQ15ygFBGh1Kt4YsrPE3bekSffdJ7AtGIBIjwKb4Z2MLULrgh+kpYmJiAgYWqKpKfHzXephUV1ezZs0aCgoKUFWVkSNHMnv2bOwhVlfoi1jJfV1ALy6m5s9/QTY0ND2Oy/p6PFu24vr88zBL1zZGbS0li642FYZhgNdLw8efUHr9DUhfqYQ1B07RLBIXRYBDFTy6eALxUQ72F1QFnLvBo7PjeEWb55eGhLwqIvaX4j5W0WYbznaZ/nWwB0iec0TDkP5deVREOVCnZYC91VfVrmKbG3xlBuCprSJYTdDSsgPoaNhQGOKOZURDPNGGHVQbIrntumPt/a+EomC7YmRLmVUBETbs5/WMacrtdlNcXEx1tX9f9fbIzMwMmCuiKEqXVhkul4t33nmH/Px8pJRomkZOTk6vdEPsSayVRhdwr12HsNmQrSpzyvp6Gj79jMjLL+/R82snTlD9hz/iWbMWJTWV2Pu+ReSi0JKO6v/3P4yGelNhNOL1ohcU4F67js+js3lx7TH0Ztl/ihBcOjGDBZPMVpppCREoAZ7QHDaFgQHCchtxVbt575HPqCutx9ANFFUhLjOWq39zKY6oThR8GzwXLvwVfPGoaZKS0mxodPtnoPT8ak9KiauyBFtkNPaI7k8QtF02ApEUibY+H1xelCEJ2C4Zhkjwj+zRXPUcWvo6JzZ8hj0yBhEkAqmgaivj065FscUgGhWLAiLajtJGHoO0Cypdx4l3DEJp1khKN7y4ktw0SqQMjMN21WiMvaeQVW7T+T1nkJ/zu6tIKdmyZQu7du1qKp8+YMAAFixYgNMZ2rmEEFx99dUsX76coqIiwOzEd8EFFwTsChgqOTk5fkUODcOgrKyMkpISUlNTOz13OLGURhcQsTEQKF5eVVG6uKxtD62ggFMLLkfW1YGuoxcVUfHA99Byc4n9dvsRQ9qBHKhv8BuXuo525DDPV8kWuRRg1qVauqeYh68aj92mMGt4CrGRdlxevUVmuU0RLJwS2DQFsPofG6kuqsXwLWN0r0FFXhUbntvGefd1snjf3O/DlLvg+CpwxkP2+b2iMIp2rGHTMz/FVV0OUpI5/UJmf+M32KOaRTWd3ANb/gE1hTBqEUy8DezthXKWYyYErkAIBdvMS7DNfAAIbm7TPC6W/OQWak+eQPeYYc1CtSFUG1JvefPSpYeVRx9n+pC7SYkYBkKgjEzCvnB0iyio1lQc3cu24lc4N+s+VOHApjjQDA91nhIOnFjJeeXn4X1jD7K8wVzkOG3Yrx2LOjQx6Jxd4ciRI+zevRtd15sc2SdPnmT58uVc3oGHtqioKBYuXIjH40HXdSIju17lubS0NGA9LSEEFRUV/VZpWOapLhBx/vkQqC6/3U70LTf32HmllFT98rEmhdE03tBAzZ+ewqhv30FqnzABEeApSqgq9lGjqQzisDSkpN5j3oBURfDPr85kwqAE7KrAYVMYnBzFX798jl84bpOMhiR3w4kmhdE0r2ZweFVuu3K3SVQyjL0Whl3UKwqj4ngOq/9wP/VlxRheD4bmpXDrclY/+e3TO+15A56dZSqNA+/AJ9+Bf80ETwDHfRMe4MvAEsyyI3XAx8A9NHUmCsDx1R9Qeyq/SWEAprKQEMhMVe8tZ9Opf+N89DycPz4Px80TETFtr/SkYVCvl/HZwZ+wo+gV9p/6kM35z7L86O/QDC+e57cjS+rMyCqvAbUevK/tRlb1TGHGXbt2BXyaz8/Px+Xq+DkdDke3KAyA5OTkgEUMpZQkJCR0yznCgbXS6ALC6STl5Zcou+MupK8EgtQ0En79GPaxY3vknN6cHMruvAu9oNA/rAlAVdGOHMExcSJS15G1tYjYWD8zReQ1i6l+4knTtNaoeBwObEOzccyZzYQjW9lw2D/ZLjHaQVyzcNr0hEieuXsWlXUevLpxOtejDYKZxI02WrR2N9KQFO0/havKTdqYFKLbMKcF48CHz2FoLZWroXkpPbSDmuLjxKakw/v3mOG/jXjroPwwbH0G5nwvyMzLgUrM0ulNBwJFyNr1eJcnYxwsA5uKOiPDNPsoCgXbVqC7/VePQlUQqoLh9bQat5E1/YKgJqxAJA2fgGKzo8k68qs2N42rzkjGTroBDur+TnJDou0owt4NPddb09Dgf71g+iM8Hk9Yk/PGjBnDzp07W6w2FEUhKSmp364yIISVhhDC7z8daOxsxTF1Kunbt5L0r2dI/MtTZGzfSvStt/bIuaTXS+lNt6DnFwS980qvFyUlhao//JGiseMpmjyV4qnTqXv9jRb7KVFRDPjoQyIWLACnExEVRdQN15Py1psIIfj2gtFEOlSaWyqcdsmcmTnc+cmXuOGDa/nl+p+TX5MPQEK0IySFIRRB5sQ0PxOIUARDZvRkq9XTVBXV8Mq97/Lpr5az4s/rePXed9n4wvYOO+Nrio6b4cqtUGx26koKoXBrYPOl1mBmrwflEOC/WpRugfuZBoydxVDnhSoX+qrjeN/eB0BkQnJABWB4PX4KQ3VE4IxNYOJN323zGlujqDbmPfAnVGckqsNcTdqckaSNm8mA7GmBP5e6NEOHe4BgBRZtNhsxMeEtBhkREcE111xDVlYWQoim6Kkrr7yy24pChoNQVhpvA9Najb0FBC53ehYi7HYizuv5KB336jVmaG8wnE6c8+dR98qr1D39DzOqCzBKS6l69CcocbFEXnFF0+5qZgbJzz4TcKoRabE8d+9s/rPyCPsKqhmYFEX04I/ZWrkdj69MyLaTW9lftp+/Xfw0yZGhF4E7777ZvPPDT9DcGppbxxZhwxFlZ+49PV8mXErJJ79aTm1pXYsn4j0f5pA2JpXsmQNDnmvAuBlUHNuLobWsuaV7PSQMGQ0NBWb/70C0WS8rG7OUesunaG3HLHDZWlqovAbGoXKM0pcYsWAZx1YZ6O2kQqiOCCbc+G1GXHwjjui2w3YDkTZhFlf/dSl5az/GVV1O+sTZpI49B1lWHzA6GLvSpZInbTFt2jSOHTuG1+tt6uNts9mYN29er5YhCUZ8fDwLFy5EStmvFUVzgioNIcQYYDwQL4S4rtmmOMAqyBIGjMrK4LYdIYi4+CISnnyCk+fMalIYjciGBqqf/GMLpdEe2akxPHaDWUDwVP0pvrl0a4uihBKJ1/Dw4dEPuGv8l0OeNy4thlv/eQ2HVx2j/HglKcOTGD4/G7uz562lFXlV1JXW+5lQNLfG3o9yOqQ0Rl95J0eWvYWnvropCk11RjL8ohuIiEuC2ESIy4KyQ7Q4oT0aZn078KQAXAL8BbPbX6OGUJF5I/yT/AAUDVm8lsQJbmbe62Dzsx4QAq0hyGpUGmTPu6pTCqORiLgkRl1xe4sxkRKNMiYFI6f0dI6GTUEkRKCM7RlzTExMDDfeeCO7d++moKCAuLg4Jk2axIABwSsyh4MzRWFA2yuN0cAiIAGzjnMjNcDXelAmiyA4Zs1EBupTHBlJwi9/QfSXbsOoqGjyr7RGL+h8M6S86uPYFbtfJVuv4eVgRU6H53NE2Rl3+aj2d+xmPA1ehBr4C+zpYLZyZEIql/3uLXa9/hTFu9biiIln9JV3MeKSm8wdhIDbPoTnLwZXJSBAd8PM+80oqqBEAP8FfgNsMo9jHiJ5LqhlAZL9NET8KQCy59sZNMtG+VGDtX/20FAWwHEuJWoP9bK2XzsWfXsR+pZC0AyU8QOwzRkYcmvVzhAVFcWsWb1R4t4C2lAaUsr3gPeEEHOklOt7USaLINiysoj5ypepe+FFpC9CSkRGYhszmqibbjRfx8ejxMRglJf7HW8f0/neBJkxmWiGv8KyCRvZcdmdnre3SR2eFHC1pjpUhs9Nh4pciEkPISTWJGZAFnPv/33wHZJHwgO5ZihwfSkMPhdiM0KYOQP4K+bSQgAqtuku9E0VLSLmUAQioQwx8HizaxGkjlEZfaXK7tcdLaKphKKSMmoqzpiEkK6vowghsE3LxDYteMi1Rf8mFHtAmRBiGZAmpZwghJgEXC2l/HUPy2YRgLifPIpj1izqXnwJWVtL5DWLib75JoSvLIFQFOIeeZiqn/28hYlKREQQ9+NHOn3ezJgsxqdMYE/p7harDZtia1H6vPcxMJ3GkZjNjtpGtaucd99sVv55PbqmIw2wORVmZb/P+P1fg/0AEub8AC78ZWBHdkdRFBh6QScP3gE8DxQj4qfjuP1mvO+danIsi6GJOBY/F1DM0VcMo/TgEIq2r0T4QpAjElKZc/8TnZSl5yktLWXz5s2UlJQQHR3NtGnTGDr07I67cblcbN++nWPHjmGz2Rg3bhzjxo0Lm89GtFsSQIiVwI+Af0opp/rG9kgpJ3T55EL8B9MEdqpxPiFEEvA6pjcwF7hJSlnh2/YIcDdmLOJ3pJSftXeOGTNmyC1btnRV1H5Hw4cfUf2HP6AXFmEfM4a4Rx/BOXNml+Z0ay6e2f1PVpxYjmZoDI0fxrem3MeoxM6vYDqCR/dwsr6YRGciMY5YzHiMpzHzGCKArwB3Eqx0RnPKcivY9+kh6svrmZK5hAEFTyGah8bao+CCX8K5P2xfMM0DhtcsWxIKUsLRpXDoEzO3ZNIdkBCoJMgHwP8BjSsFG6ZyfAVZl2z6C5w2TBPW9zB9II04gceB+VTlH6b86F6ikjMYMHZGh0Jse5OysjLee++9FnkXNpuN2bNnM25c29V8O4OUkt27d7N79248Hg8ZGRnMmjWLxMSeSUTsDJqm8eabb1JXV9fC0T9kyBAuvvjiHj23EGKrlNIvOiUUpbFZSnmOEGJ7M6WxQ0o5pRuEOg+oBV5opjR+D5RLKR8XQjwMJEopHxJCjANeBWYCmcBSYJSUMkh4isnZqjR6EkMa6FLHrvRe0bV3D73DKwdeQiDQpMY9E7K4fOhOhGgeTRYBfBP4UscmfyIN6k75j0elwoMBxhtxVcOH34T9b5nl2FPGwuK/QtYu4DPMlc9i4HbM3qaY0VSvLobcFWbOhuoAocINr8KYxc0m14BLMV2IzVGBhcDPWo1vx1SgR4HBmO9D29Vq60oKMHSNmLTBfcJR++mnn5KXl+c37nA4uPPOO7v9yXrNmjUcPHiwhZKy2+3ccMMNXe4Y2F3s37+f9evX+yUwqqrK9ddf36NJgsGURijmqVIhxHAa80qFuAEo6g6hpJSrhBDZrYYXAxf4/n4eWAE85Bt/TUrpBo4JIQ5jKhDL39LLKEJpUXeop1lTsJqXD7zYok/41LQtCNH6ecEF/Ae4jVBWG03UB2k61BC4kyBATVEue39/DWVlNcTaUxmfUkYyu8F+H8gYEI0mvGeBzcDfTJn2vH5aYQBN8bH/ux1+VNLMl1JA4FAp3Tdfa6YCgcOnW1NdcJQ1f/wutSdPgBA445M497t/JGWkf6vd3qSkpCTguGEY1NfXd2veRUNDAzk5OX5lPjRNY9euXZx77rnddq6uUFhY6KcwwPQdlZSUhCWzPJRv/n3AP4ExQogC4AHMx5ieIk1KWQTg+90YO5cFnGi2X75vzA8hxL1CiC1CiC3BPogW/Yc3c15voTAAkiKCLTCraZlJHQIpQUwfKYGz+ivzDvLpQ9dyrNBFtdtBQW0My44PpjBhAMTbmikMME1Gu4E95stdLwXu/ZEdDfU/AJ4A9gHxBFYaAClBxttH93pY+os7qMo/gu51o3tc1JcUsvzXXzXrZ4WRYEpBStntmd2VlZVBS3ycPHmyW8/VFeLi4gKusIQQREd3f3HMUGhXaUgpj0opLwFSgTFSynlSytwel8yfQI+OAW1rUspnpJQzpJQz+nO6fnPyy+v50ycH+MHL23h5bS61rsBhtWci5W7/MusFtcGc3gPocHWcy/+f6cNoji3SHA/A9peeQHO7OP2RFOhSYcvuBKQj0FdKB/aaf6oBQl2vHgw3ZELcRuBN4F5Mf81coPX+EcBdoVxVQAq3rUD3umn91ZGGTu7qDzo9b3cwbdo0v9aqqqoyevTobm+5GhsbG7SYYF/yaYwdO9ZPaQghiIyMJCMjlCi87ieUMiLfF0J8H/g68DXf67uFEFN6SKaTQogM37kzgEajcj4wqNl+A4HCHpKhT7H1WBm3/30tb23KY+3BEp5Zfohb/7qW8lp3+wefAYxNGnu6fLeP5/bE4tZbP0dEAN/p+AmGXwJ3LIGhF5vhttkXma+HXxpw99Kc7QHH62sE3qpABQVtQJr557R7zOS+RgZFw4QkcCg+HWRw2sz2LUwLrAOI9l3f1zltvW2NxHSeXwecj2kkaJlDU19+0i+DHUD3uKkv7Rarc6cZMmQIc+fOxel0oioqqlAYlTyIOVPb9s10hpiYGAYOHOi32lBVlcmTu99M19l+MTExMVxxxRXExMSgqiqKojBgwAAWLVoUNj9UKOp7hu+n8TFkIaZR9RtCiDellG0EqXeK9zEfpR73/X6v2fgrQog/YjrCR2KGjZzRSCl57J09LcqUu70GFbqHf684wo8WdX9USV/jjnF3srNkB27NjeHLkN5fHsf+shuZMmANcBzzI/FNzJtlJxh8Lty1NKRdnbEJaC5/E5OCxOYXGyCAKMBXZmbUQpjyZdj+H0DCuAH+DZaajtsO/Ako9f1k03Yxhv8Az3E62mojZmXc/9LYii9l1NSm8Nvm2CKiGDCu58u4tMeYMWMYWmCnbnMuDkPBVqri3b8RbhiPOjL0UjWhcPHFF7Nu3ToOHTqEYRjExcUxf/58kpK6r+RJXl4e69evp6qqioiICKZOncqECRM6dMPPyMjg1ltvpba2FpvN1m1VeDtLKNFTnwHXSylrfa9jMNfO1wJbpZSdvmsJIV7FfGxKAU4CPwfeBd7ADAHJA26UUpb79n8U+CqmsfcBKeUn7Z2jv0dPnaxq4KY/r2nRUrWR1DgnH/zggt4Xqidw1wISnIGjVgpqC3j1wMscKN9PWlQ6N42+hcmpPei4rciFbc9CdT4MXwDjbgCbaSo6+Nkr7Hj5yRYVZVVhMGRwEgnn3s2g894hIr4cIUCIEcDvMBfGzSg5YIbdDtkPaVvAz6kfCTxIy2IMbeHCLD/SujaZAC7CDN01Wfl/3+Tkng1NSX8OZyyjBi9k1KjFiHgntnMGomSGJ3rIOFGF56Wd/q1i7QrOH5yLcHR/yXvDMNB1vdtbsBYUFPDpp5+2MIPZbDamTJnCtGmty/n1PboScrsfmCyl9PheO4EdUsqxzcNw+yr9XWlU1XtY9OQKvAEqwWWnRPPa/fPCIFU3UnEM3rkT8jear7NmwrUvQNKwXjh5LebzyReYzY1uBs6HQ5/B69eZkU1SN/0QyaPgno3giEJKyc5X/sjBT15EsdnRNQ8J2XM4kXMuUlcwdElcRgMDRg7gwgeuQgnUc6WJ45jRXq1NjRHAJ7TVdKkluZg5KoF6qWRiLtRNDM3Loc9f48iyNxG6wrlp38JhRJg9MARgU7AtHIVtUnqI5+4+vB/loG8NYCZzqNgXj0HtoRpWPcG7777LqVP+Idt2u5277rqrTxRUbIuuhNy+AmwQQjSaia4CXhVCRGOGeZx19GbFyvgoB5OHJLIjtwKtWb+JCLvCDTPb7hHdExRW1FPn1hmaGo2tzZthCHhd8OwcqC8x8xwA8tfDv+eYpTcC9f3uNhowb7LFmE2PAHaDcQu8/XOzfHkjugdO7YGlD8GVf0EIwZQv/YBx195LbXEe9uhk3rh/ObpHp9HBXF0USX1FA8fWn2D4vLb6Yg8BfgA8yemvo4Fpne3I034KwaOtWq5yFJud0Vfcwegr7sC7Khd9TZ6pMPCJ7zXQPj6EOm5At9SMqistYv/7/6bkwFZi04cwdvE9JA8PnBssA5bJ9dGL/Va6g6qqqoDjhmHgcrm61Eo2nLT5iRDmnfG/mAUKK4Eq4BtSyseklHVSyg5mUfVvXEuXUTzvPAoHDqZo0hRqnvlXpx1cHeGx6yeRnRpNpEMlyqnisClcPD6d684Z1P7B3URxZQN3/WMdt/51LV//90aufGIFqw60kfgWCvv/Z4afymamCGmYTYv2vd2hqTS3RlluBQ0h9234ADPGonmRQheIl8AeuLEPW//V4qUjKpakYeOpOKEjAihQzaVxeNWxEGS5DvgIeBj4CWZiYEdXkDHAFZiZ4M2JwPRrBMY4UHpaYTRHgDxZ20EZ/KkpzuOTHy3m8NLXqTx+gBOblrDsF3eQv+WLgPur44P4eAyJMqzvRDWFQrAcCkVRwtocqqu0udKQUkohxLtSyunA1l6SqU/iWr2Gsq9/A3z9LIyyMqp//wSyoYG473YiYqcDJMU4efGbc9lfWM3JKhejM+LITOw9Z5iUkm8/v4XCivpmD3s6P3trJ899fQ5DUzuZdFVxNHDbU08dVLa82VYV1VByqJTo5GjSx6W2WOntfHcfW17ZhVDMtrGDpmVy0ffOxR7Zlo16Df72f0DaIdMJOQEUh+6B2lMQ07LstmJXggR/m4UQQyMRuDLEfYPxCKbT/R3MVUcS8EP82+E0IzLILUCXENH1MNddrz+F1lCHbHwwkBLd42LLs78ga5p/10BlWCLKuFSMfSWmX0MRoAhsC0ch2vx/9j3OOeccPvnkk4A+jb5ummqLUD4VG4QQ50gpA6WhnjVUP/FEk8JooqGB2r/9ndhvfbOpYGBPIYRgXFY847Laat7TM+w+UUlZrdvPOuDVDd7edIIfLuxka9uMaeCIAU+rUhmOaEg3XWWGbrD8qXUcW38CRRUgISo5iqt+dQnRyVEc23CCLa/sRHOf/mKe2FbIij+v59KHzmsa89R7OHmgFHuUnbRRKQglDXOh3eopWwhoCNKHWygB8ywyxqeZsrXC5lQZc8mIkN6K7sGGaer6LqZvI5b2MuNtswbiza9u6XgWIJIjUZK7bj45uWfDaYXRDHdtNQ2VpUQltVTAQgjsV49BTstEzykFh4o6MQ2lFx+SuovMzEwWLFjQFD0VGRnJlClTGD9+fLhF6xKhKI0Lga8LIY5jVoYTmIuQST0qWR9DOxrYzCA1DaOiArWPNX3pTspqPSgB7j26AcVVQUw5oTDiMkgcBqUHzD4TYN6UE4bCSLNZ1P7PDpG74QS6R2/K864uqmHpk2tY/LsF7HhrTwuFAaB7DY5vzsdd68YZ42TfpwdZ9++tqDYFKSWOaAdX/+YK4tI/oeVqQwGRDBlfgbyn/OUdPA8iE/yGVZvC5T+5kI9/+QUgkYZEGjD+ytEMnBKOBCwbZq+09lFHpWDMHYS+9gSoAgyJiHPiuGVit0jijE3EHSjTXBrYowJnNAshEIPiUQb1/gNSdzNo0CAGDeo9M3JvEIrSCL3V2xmMfcRwPJv9o7CE3Y7ShzJIe4LxA+OJrPUS5dEpj7Sh+zRIhF1lzojOl7RAUeErq2DFz2H3K2b114m3wYWPmduAPR8f9FMK0pCUHCqlocpFfWVgpSUUBVeNh6qiGtb/Z6updDzmPF6XxnuPHOP2f/8UofzWd4SO6TD+f3BpMpQdhGOmEkCxQ0waXP9y0EtJH5vKHc9fz/FN+XgbvGRNziAuLbw9qkPFfv5QbOdkYRTUIKLtiIzYbgv0GHPVV9n63K9bhCcrdgcDz7kEe0TPlcGQUiLzq5HlDYi0aJT0vlGA8EygXaUhpTwOIIQYwFnc5jXuwQcpu+POFj26RWQksfd/u8dNU+GktqSOlb9ezqL8arxSIqRkQ1YMeQOiSYl1cuWULjbbiYgzy3UEKdmhuYJEBAmB5tZIGpJAbYl/mKliU4gdEM22N3c3KYsmJHgbvBTtn0Lm+KXAQcyM62xzuw24/WMo2gGFWyBhiJkt3o4d2u60MWJ+dpv79FVElKPbk+cAhl1wLTXFuRz86AUUux3d6yV9wmxmfv2xbj9XI7LBi+fFnciyep9dBMTAOBy3TETYuz/P42yjXaUhhLga+ANmsPcpzBjB/Zj9w88anHPnkPTvf1H1y1+hHTmCkpJC7HfuJ/quO8MtWo8hpeSjXyyjqrAGYcimKkhzC+o4f+4Qbr1uApGOzjlLXZqL9468y8oTy1EVG5cNuZwrhl6J2ipbeejsQez95CBGqwifyPgIhCIo3B2guJyAOV+dhqIquKrdQdqqCzy1HsyS5UE+yhlTzB+LTiOEYMqt32fc1fdQlX+EqOR0olN61mTn/egg8lRdixBdeaIKbUUu9kuH9+i5zwZC+cb/CpgNLJVSThVCXAjc2rNi9U0iLriAiAsuCLcYvUbp0XJqS+uRrTzgqpQMK6ojJqJzKyzd0Hl49YPk15zAY5ghr8/ve44dJdv5yWxfn4jKPKg6zrSrhnFs4wlcVW40t4ZiU1BUhQsfmMu+Tw9hBIjrV2wKyUNNk+HQ2YMp3FXsZ+IyNIP0cWH0Q0kJrirT6a+euSvVRhzRcaSO7vk8YGkYZhhx66gNTaLvKLKURjcQitLwSinLhBCKEEKRUi4XQvxf+4dZ9BZGRQWVP/8lDR9+CLqO86ILSfj1r7Fldc105KpyowTygEuor+i8A3xj8QYKawuaFAaAW3ezs2QHR0t2MWzJj+HYMlCdRGgubrn6Hg6o91G4u4S49BjGXT6K2LQYdr+/328FAqZjurqoltThyYw8P5u9H+dQmV9lKg4BNofK9FsmERHbOqehl9j/Dnx8P9SVmL6baffAgiebypR0JzVFuRxe9gb1ZSfJmDyPIXOvRHWE6bp7A4OAPeCBwPkoFh0mFKVR6as3tQp4WQhxCjh76nL3UVyai09zP2FdwVrs2/Zw/sECxrvNCCT3si8o2XEVaWtXo3Qh63TAqGT0QDdlh8rgGQFbmYTE3tI9uHT/HAkpJfZPvgd560BzmT+Auus5xl8yhvEPfbvF/mmjUzmxvcjPZ2HokqTsBPNYu8rixy/j4PKjHFt3HGeMk3FXjiJzfFqn5e8SuavMhkuNrWUNzBpX3npY/Gy3nqpg63LW/un7GLqG1DUKty7nwIfPcemvX+1RJ3RPULRjDTtf+xM1xbnEpmcz+ZYHyJjinwApbAoiMxZZ0CqMW4Ayovt9NmcjoWSY7MQM+v4e8ClwBDjQk0JZtI1bd/Ojld/npf0vcqBiP7uHqjzzlSw+usJnbtF1ZG0tDe+/3/ZE7eCMcTL95onYnKf9DKpdJTopkrELRnZ63pTIVByK/1N1BJCVu7pJWTThrYcN/o7yMQtGYI+wtUhFUB0qA6dkkDjwdLimzaEy7rKRLPzlJVzyo/nhUxgAq359WmE0ojXA7pdNc1U3YegaG/7+CLrHhdTNYALN3UBNcR4HPw0eBdYXKdi6nNV/uJ+KY3vRGuqoOLaX1X+4n4KtywPub79qNDhVsPk+GHYFouzYF1imqe4gpDwNaWbnGJjtVxFC7OpRqSzaZHneFxTXF+Np1s3O41T59LIBnLe6jNhaHVlfj3d/13X71BsmkDwsiT0fHKCh2sXQ2YMZf+UoHFGdt8NfOPgiXs15pUVenUAQgxI8Fa3BvxFTRKyT6/5wBRue38aJrYXYnDbGXjaSaTf61zWSUpK3tYBDy4+BgFEXDWfQ1Ize70lQfijwuGKHmiKIaC83YStmnarDmLkYXwK+TOvnv8q8gxgB2oQaXjd56z5h/DX3dlTysLH9xd83VeRtRPe42PbC/5E1/UK//ZUBMTi/PQttWxGypA6RFYttcjqikz44i5YEVRpCiG9idoEZ3kpJxAJre1owi+BsLt7k1/4UwKZJjg6LZvKuakRUFPaxY7rlfIOnZTJ42mn/SGFtISdPFTM4djDJkR3P00hwJvDY3F/xxOb/o9pTjZSS9OgMHj7nEcTR7X4lREBAduA+GbEDYrj0R+f5jVd7qll6/HNO1OQxMmEU9vdiydtQ2BTCe3xzASMvGMp535zVYfm7ROY5UJXXst4WmNV0E9oqbAhmfdDvcjohsQqzh0YVpiHgNDZnJNII3PbWHtm/CuXVFB8POF57Mi/oMSLagX1+e++nRWdoa6XxCmZt5t9hVlJrpKaxv4VFeEiMSERBaWpI1IgUEFOrgaoiYmKIvPrqbj1vg9bA45t+w97SvdgUO17Dw/ys87h/2ndRRcfi38ckjeXZBc9RVFeITbEzIMpnWrvqn/DaNaaJShrmE7g9Ci4JPfbiRE0eD676IV7di8fwsGv7HqavmY+qnf64ay6Ng18cZfwVo0jO7sXkzAt+Doc+btkn3B4N8x4Ooarvv/Avoe7CbG9zL2auiUlc5lCiB2RRXXCshYJSnZGMXHBb166hl4lISMFVUeI/Ht+FxFKLThPUpyGlrJJS5kopb5VSHm/2YymMMHPF0IXYW4VpCgOi63WGnvAQcfHFpH74QZec4IH4586n2VO6B4/hoV6rw2t4WZm/gp+v/SmHKg52eD4hBJkxWacVBpgtVu9eb2aGZ86Ac74J39oNKaNCnvev2/9Mvbe+KTor/kQyiub/UZe6wYltvdwxeMB4+OoaGHYpOOMgaSRc+Rc479EQDj5C4MqIKqe7Ip/mvB/9jaikAdgiorFFRKHYnQw7/1oGz+1qYcTeZcJ130R1tlSoqjOS8dd/I0wSnd2024Spv9PfmzAF44u8Zfxj599RhIIhDZIikvjZnF+SGeMfZnuyqoG3Np3gyKkaxmclsHj6QHKKqtmeW05qrJPLJmWSEN12uKdmaNz84Q14jcCBc07VyXkDz+fbU74Ttt7FAF7Dy40fXIfR7Ok6e+9oxmydhKq3XFjbnCpzvjKdcVeErpDCy/cwq/O2/s46gc8xK9y2xDB0Tu3dhKuqjJRRU4kZ0Pmot3AhpSTn4+fZ8/bT6O4GU2Fc9w3GLPxyWD9rZzqd7tzX3zlTlQaYUVSHKw4RZY8iO25owC/QgcJqvvncJjTdwKtL7KpAN8BhE7i8Bk6bgqoInrpzBhMHJQQ8j6EbFB4u5uH1P6I6vjJo4dQINYIfz/oJUwaEr5mjbujc8MG16PK0Pd9ZH8GFb10VUGnc9uy1RMZ1tTpOPWZAYTyNvbh7hv2YrW2aO4UjgBuAB3rwvH0Dw9DR6muxRcWgBOhzbtG9BFMa/beo+1lOWa2b97cUs3V/FFpDStAnrt+9v4cGj97ULtarSwwpcflKYbs1g3qPzqNv7AjYUCpvWyHP3/km7z/6OXM+uIQL3l5ETGXgCqou3cXKE4HDIHsLVVGZlTG7hY/FHeVi14UbwSGxR9mxR9qwR9i49OHzu0FhvAkswFwF3AXcgtkNsCcYCzwFjMTU3PHA3UDP9nMxw9xewrzOWcDtwPYePqc/iqLiiIm3FEaY6XqXFYteZ9WBU/z0zZ0AaIbk3yuOsHBqFj9aOLaF8nB7dQ4V1wSbpgXVDRq5pXUtGirVnKrl88dXorl1VFRARa1Rmf3JxSy7+V2k0lLJCASigw7xnuBbU75NQW0+J+tOIn2mnIQpsdz+1Wsp3Vdp+lImpmFzdvXjvw3zJt78yf8oZoTTa7TXy6JzTAde7YF52+JvwOucvs4DwP2YjvlO9lKx6LdYSqOf0eDR+Plbu3A3y9TWDcknOwq5YGwaM4ebWa8nyup47J3dHWirLFFblQzJWXoEXW8ZoSVQUHSF1PwMTg1u6UR2qA4uGnxRh6+pu4lzxPHnC//G3rK9FNYWkB2fzciEUQghiD6nO8uVv4p/9z8DKMB0WvdmA6aeogFTAbaO2nID/wT+1NsCWYQZS2n0MzYfLQ9YobvBq/PJzkJmDk+m3q1xz7MbqW4IvdpLSmwEg5JaOlLryhuQmr/WEVLgdEUgEDhUB7rUUVC4PPtKJqRMRErJsaqj6NJgWMIwVKFSXuvm/W35HDlZy/iB8SyamtXpgoehIIRgQsoEJqT4J/p1H8ECCVWgsgfP25ucxLye1kjMBEOLs41+pzSEEJdj2gRU4Fkp5eNhFqlXaStwoXHbsr3FeLxGwLptqgLxUQ7q3Rq6IbGrCjZV4fFbpvj5RQZOzeDw6mNorpZJYkIKqjMquHbE9aRHp9OgNTAjfQaDYgdzpPIIv9n4GLWeWrN1p2LnSyPu5w9vNeDRDTyaweqcUzy/+hjP3Tub9IReaOPpqYeaQojNBEd3hiHPxzTVtH4K1+iS2cZTD4c/NXNVhl8K0amdn6vLDAACJwnCsN4UxKKP0K+UhjAN5n8DLgXygc1CiPellPvCK5lJnVvjpTXHWLK7CIdN4doZg7junEHY1O6LNzhnWDJ6gGKdkXaVK3wNkfLL62nw+n/RFQG3zcnmvgWj2VdQxc7jFaTEOpk/ZgARAZrTZM8aROKgBEpyy5pKVGo2L8XD8rEPULlh1A3EOE53RHNrLn6y9hHqmiWuNdDA03t/T4N+J7pmJp+5vAYezcOfP8vhtzdP6cK70Q6GAV88ChueMqvJGjrM+g5c/Nt2GyqFxg3AO0AppxVHBGYhhQ4UBGyohBW/gL2vmzK6qsy2twIwNLj09zDr/m6QtzNEYV7n2/hHbfWfUiQW3Ue/UhrATOCwlPIogBDiNWAxZn2FsOLVDO751wbyyxvw+u7qf196kK3Hyvm/W7svBDXKaeMX103k52/vQgKabmC3KVw2KYOZw0x/xuiMOCIdKg2tqr867SozfD6PcVnxjMtqu86RalO4+jeXsu/Tg+xcto8KvZyS8QWMOn8ED458oIXCANhYvBHD8NdoUho4kw5Qf2p605ghYf2h0s68BaGz7knY+GezIGAjm/4CkUkw78FuOEEM8DJmBNUqIAmz1YxflGJwNA88Owsqc0E/XSqeZmXj+fwhGHI+pE/qBpk7w3cwqwe9DNRgrjB+APSk6c+ir9LflEYWcKLZ63zMGMAWCCHuxfcYNHjw4F4R7It9xRRXupoUBphP1BuPlHKouJqR6YHDVENBLy2l5qk/41qyBBETy4y7v8rb372GZftOUe/WmDMylTGZp+c/b8wAUmOdFFU2NIXaOlSFwcnRnDP0dHloKSWV7grsit1PATRic9qYtHgckxaPa1fOandVi/yIRoSiI2x1fuN2Ww9HfK97wr+irLfeHA+iNNx1Ho6syqWquIYBI1PInjUQtc0WoTHAV3w/nWD//0zTWXOF0RrdAzufh/Q/dO4cXUYF7vH9GFiR+mc3/U1pBIph9LPcSymfAZ4BM7mvp4UC2J5bEdAkJKVkb35Vp5WGUV3NqcuuwCgrA69pI6r++c+J2rWLmx//XcBjbKrCs1+bzT+WHWLpnmIUAZdPzuRrF45oaqqUU36AP237I6fqTyGRjEsaz/dn/JCkiKROyQkwPmViwP+QIh0Y9S2Vt8OmcNXUHs5ObgjiqA4yXpFXyXuPLEH36mhuHVuEjehXorj295fhjOmhxkX5G8BT2/Y+Um9/n17DUhhnO/3tE5APDGr2eiDQy8WDApOeEIFD9b9jKvX1KA/9gNK7voJ3//4Oz1v38isYlZVNCgNA1jdQ98ab6IVFQY+Li7Tz4KJxLHn4Ij596CIeuHwM0b68hLKGMn669lEKagvwGl40Q2NP2R5+/MUPcO/fhwxgYgqFofFDmZMxF6d6OmHOqToZlTSSgY7xRNoVohwqEXaFSYMS+NpFPRySmhqk93eQ8eVPrcNd52lqDau5NGpO1rL5lR7sBJA0wizI2Bb2aBh7fc/JYNFtSCkpKSnh6NGjVFdXh1ucHqG/rTQ2AyOFEEMxg+FvAfpEyc6rpg7kv0sPgDj9lgpDJ9LrYkrORtw5kpJ160j98H3so0eHPK977Vpw+Xe5Ew4Hnt27iMzM6LCsS45/5mdGMqROWXUx6x64jTEnBUl//SvOeed2eO7vTf8BK/NX8Fnup2iGxrnG+WjvqJTn5qLYFKKmZTDtjimMHdwLlWWveApeXtjSp2GLhMv/5Leru9ZDWW6l37rV0AyOrjnOvHvP6RkZJ30Jlv/U34zWiD0aRl5pRlFZ9GlcLhcfffQRVVVVCCEwDIPs7GwuvPBClG4JvOgb9KsrkVJqwLeBzzAL8bwhpdzbE+fSCgrRCkJfxCTHOvn58r+TWlOKw+vGrnkYWnaCX334f6jSACmRDQ1UP9Exu7RtyBBQA9jUdR01o+MKA2haYfgjKYs2MEpKKfvKV9tcyQRDEQoXDrqIx+f/np+O/AXl/2ig/FglSDC8Bg3bi8l7uZd6eA29EL68HIZfBnEDzd9fXg7D/BMQhULgArKACNQnvbuITISvrIK0KWbElOKA9Kkw+S6Y+lW46U248XWwCvP1eVasWEFFRQWapuH1etF1nePHj7N79+5wi9at9LeVBlLKj4GPe2p+74EDlH/jW2h5ZoMX2+DBJP3j79jHtN/QaExVAU+//jAnY1Ow6xrJ9ZUtd5ASz/aO1eyJ/spXqH/9DWRD86dlG2r2EOwTJ4Y0R1mNmxfWHGXDoVISY5yMHzcYp+r0a+RkKIIheeYTr9Q06t54nbgHHuiQvM3Z+e4+v/7dukfnxNZCakrqiE3thT7VA2fBHZ+2u5sjykHamBSK95cgm6XRq3aFURf1cD5C2kT45naoLwOhQmRCz57PotvxeDzk5+f7RQ9qmsa+ffuYPHlymCTrfvrVSqOnMerqKLnuBrRDh8DtBrcb7dAhSq67AaM+iPmgGZE33oBwOkmvKfVXGD5sA7PYnlvOw69t595/b+SF1Uepc/m35WzEPmI4Sf/+F0paGiIyAhwOHDNnkvLqKyGVhS6vdXP70+t4e9MJjpfVs+N4BW8vi0OVUdiamdLsboNx+2rILPIpEo8HvQMrrUCUHatscQNuRLUrVBeFVhOrN7nwe+cSlRiJPdKGYlOwRdhIGZ7EtJtDU85dJirZUhj9FF0PlgAJXm/olRn6A/1updGTNHz4YQuHcxNeLw0ffEj0zTe1eXzcD76PZ9NmtAMHkC6XmVzWDBEZybIb7uMfL21tqjKbU1jNe1vzef4bc4KW1Yg4/3zSt2xCz89HREejJicH3C8Qr60/Tq3Li9bs5u3yKBTvuYnrLz/B5oLVqKfKmL+qlEu+OJ03IaKicM6fH/J5ApE6MomSI2VIvaXi0L06CVmdD0HuKWJTo7n1mWvI21JAzalaUoYnkTFugNWzwaJdIiIiiI2NpaqqqsW4EIIhQ86strPWSqMZelFxSzOQD9nQgFHcfrlrJSqK1PffJfmlF4h9+CGcF14ATidERCASE3H86jf84zhNCgPM0uSlNW7e3nQi6LwAQlGwDR7cIYUBsPFIWVOuRnNUojg/5TZeuOo1/rBzHJevq0NtFCsiAtuwYURecXmHztWayYvHYXO09MfYHCrDzh1CdHLf7FOt2hSGzh7EpKvHkjk+zVIYFiEhhOD888/HZrM1Ob1VVSUiIoIZMzqQ7NkPsFYazXBMm4qIikTWtTRFiago7FNDy+p2r1pF5Y8fRT+RD3Y7UTffROy3vomamcmOE1Wox7b5H+Orx3TXed1vO0+Ld5ITwJ+t6ZJkX+5B4p/+H/XnzqXuhZeQLhdR115D9N1fRdi7VlAwNi2GxY9fxrpnt1C8vwRHlJ3xV45m2k3tZxLXemrxGl4SI3qxf7eFRRdIT0/nxhtvZN++fVRWVpKens6YMWNwOnsoxydMWEqjGc5587CNHYd3z57TYa4REdjGjg0p/NSzaxfld3/t9GpF16l/401kXR1Jf36K+CgHepBa5Ukxbbdb7Sy3zR3KpiNlLVY3NlUwJjOOLF9VW6EoRN98M9E339zt50/OTuSqX4ceLlrhKuePW59kb9leBIIBUWk8MO17jE5qPxDBwiLcxMbGMmuWX5GKMwrLPNUMoSgkPv5blNTTVUVtgweT9PTfECHEWdf8+S+mL6M5LhcNH36EXlbGsAExZCVF0TqCM8KucPPsnrF7ThmSyA8XjiPaaSPKoeKwKUwenMjvu7EeVnchpeTHax5hd+keNEPDa3gpqM3np2sfpayhh+tUWVhYhIS10miGUVVFyQ03IZs5s7Rjxyj70u0MWLa0XcWhHT5MoHrkwuFALyhATU7mj1+axvdf2kZBRT2qItB0yTcuHsn0oR3zVXSERVOzuGxiBrmldSRE2UntcovTnmFv2R7KXGUYrRIPdamz5Phn3DrmS2GSrGO4a92cOlhGRKyTlBFJll+km6ioqGD9+vUUFRVht9sZN24c06ZNO6MS5/oDltJoRv3bb5uhts1v/F4vekEhnnXr2zVR2adMQTt6DFqF30mPB1t2NgBp8ZG89K25HDlVS1W9lzGZcU3lPXoSu01hZHrgooR9hVP1pwIqXa/hJb+mIAwSdZwd/9vLlld3odoUDEMSnRTJwl9cTGxad3YMPPuoq6vj3XffbQpf1XWdnTt3UlVVxcUXXxxm6c4uLKXRDO+BnMDRU4aOdvRou0oj9v77cX30MbJZToeIjCT6rjtR4k6HmAohGJHWt2/gwaj11LI0bwk55TkMjhvCZdmXd6nIYXNGJIzEkP51r5xqBOOTW9aLKqkv4ZldT7Pl5BYUoTAvaz5fm3hv0Gq9vUH+jiK2vrYL3aM3JTVWF9fy8a+Wc9NfFlkrji6we/duv1wIXdfJzc2ltraWmBhLKfcW1rquGfZJkyDKPxRUCAXb2PY7sdmHDyP13XdwzJuHiIpCzcok7sePEPeTRzslj37qFO4tW9HLg7UV7V1K6kv45tJ7eXn/S6wtXMNbB9/gm0vv5WjlkW6Zf3DcYKalTcehnA4KUIVKrCOWC5v1HndpLn648ntsLt6MLnW8hpfV+at4ePWDAZVOb7H7wwNNxQ4bkYak9lQt5ccrwyPUGcKpU6cC9mpRVZWKioowSHT2YimNZkRddy1KbGzLWk9OJ7YxY3DMmB78wGbYx48j9fVXyTyUQ/qmjcR89SsdfsKUHg/l932b4tlzKbv9DopnzGTfw49xuKgKI0j0VW/w3N5/U+OpaSo/4jW8NGgN/HXHn7vtHA+e8zC3jfkSaVFpJDgTWZB9Gf/vgqeItJ1uC7u6YBUNWgMGp28imtQ41VDCzpKd3SZLR3FXt277aiJUBXddG/0yLNolOTk54PdI13Xi49tuJmbRvVjmqWYoUVEM+PhDqn75GK7Pl5p5FjdcT9zDD/WqaaHqt7+j4dPPwO2mwJnA/y36IafUVJR/rCMqOoLHbpjEjGE95zgPxtaTW1rcqBs5WnUUt+bCaeu6g92m2Lhu1A1cN+qGoPvkVh3DpftX/tUNjfyaE0wdEJ7IsOzZgyg9VuFXb0vqBqkjev//dSYxceJEcnJy0LTTJXdUVSUzM5O4uL5XXeBMxlpptEJNTyfp6b+Tefggmfv3kvCrx1Cie6Gwng8pJfUvvQwuF5pQ+emiH1EQn4HH5sCFQnmdhx++sp1TVf43zZ6mudmoOQoKitJWd7vuZUhcNhGqv4JSFRsDYwcFOKJ3GHf5KGJSok5nwQuwOVXm3jMDey8EO5zJxMXFsWjRIlJSUgBTYYwaNYpLL7VKxvc21ie5r6FpTbkeOwaOx21zIluFFOqGwQfb87n7gh5uYtSKBdmX8d7hd/E0619tEzZmZczGrnQte7wjnDfwfF7e/yIe3dO08rEJGwMiU5mcGr5qoo4oO9f/8UoOLD1C7sYTRCVGMmHRGNJGp4RNpr6KlBLjSDn6/lKEU0WdlI6S3rYze8CAAVx33XUYhoEQwgosCBOW0uhjCLsd2+jRaAcOUBkVjxHgi+HVJcVhWGncMvo2jlYeZXfZblShIKUkK2Yg9035dq/KEWGL4Mnz/x//3PU0W33RU+dmzeNrE7+OIsK7eLZH2pl41RgmXmVlsAdDSon3rb0Yh8vBa4AAfUshtkuGYZs5sN3jrbyM8GIpjT5Iwm9/TdmX7mDMqSPIADfBSIfKzF7yaXg0g5JqF0kxDiIddn4+95ccr84ltyqXjJgMRiaMCssTX2pUKj+Z/bNeP69F1zEOl59WGGA2v9IMtKVHUccPQET3TEkdi+7BUhp9EOesWaR++D4Rf/s7c8oPszF1FC5f7wunTWFgUhQXjE3rURmklLy45hjPrTpqdt2TksXTB/Kdy0YzJC6bIXHZPXp+izMXfV/JaYXRHEVgHK1Andizn22LrmEpjT6KfcwYkv7yZ35nSD7eWcjbm/JwawYLJqZz8+wh2G09u0T/cHsB/1l5pEWhw/e25RNhV/nWpaN69NyBqG7w8tSnB1i2txhDwrxRqXz/yrGkxJ5ZFUTPBoRDAYF/e10B2C3TU1/HUhp9HEURLJqaxaKpWb163v+uOtpCYQC4vQZvbszj6xePRO3JvtmtMAzJ1/+9kRPl9Wi+3iArD5xkT34lb35nPk5770VuWXQddXIG+o5i/9WGBGV491QXsOg5LLVuEZDy2sDJaB7dwOUN3tqyJ9h4pJSTVa4mhQGgG1Dj0vhi38lelcWi6yiZsajnDwVVgENt+rHfMgFhPQD0eayVhkVARmfGseO4f3mGlBgnUY7e/WIfK6nDo/vbwBs8OoeLayB8UbYWncQ+dxC2SQMwjlSAXUEZkYzo5c+VReewVhodxLNrFyU330rh2PGcvPAi6t97P9widRopJUZFBdLtX/7i/gWjiLArNDdCOW0KD1wxutejpYakRONQA0eRDbOqx/YI1QVHKdy+ivqy9tscdxYR40SdnI46boClMPoRYVlpCCFuBH4BjAVmSim3NNv2CHA3oAPfkVJ+5hufDvwXiAQ+Br4rZYA62j2IZ88eSq+7oakSrlZdTeUPfohRWkrM3V/tTVG6jGvlSiofegS9uBgUhcjFV5Pw29+gRJo1nsYPTOCfd8/imS8Oc7C4moGJUdxz4fAe7fsRjNkjUkiOdeKpaEDz1d5SBEQ5VC4el97r8pzJeOtrWfn7b1F+ZDeKakPXvAw5dyEzv/5Yr2b9W/RdRC/fd82TCjEWMIB/Aj9sVBpCiHHAq8BMIBNYCoySUupCiE3Ad4ENmErjz1LKT9o714wZM+SWLVva2y0kSu+8C/cXy/16PojYWDJ27+xyT+3ewrNnL6XXXNuyDLzTScT555P83L/DJ1gbVNR5ePKjfazcfwpDmorkwUVjSU+IbP9gi5BZ86fvUbB5GYbmbRpTHRFMvPk7jF30lTBKZtHbCCG2SilntB4Py0pDSrkfCGTmWAy8JqV0A8eEEIeBmUKIXCBOSrned9wLwDVAu0qjO/Hu2h2wSRCahn6qBFtWZm+K02lqn37a3yTlduNauRK9sAg1MyM8grVBYrSD39w0hcaHHKuERPejeVx+CgNA97g49OnLltKwAPqeTyMLONHsdb5vLMv3d+vxXkUdFLjEgZQSJSmxl6XpPNrhIxCgN4FwONAK8gMcERzpdqMVFAT0i/QEZ17NIQ9QAPR+WZjW6J7gMnjra3tREou+TI8pDSHEUiHEngA/i9s6LMCYbGM82LnvFUJsEUJsKSkp6ajoQYn7/vcQkS3NISIykujbbm3yBfQHHDOmg81/kSk9HmzDQyuCKKWk+qk/UzRhEqfOv5Ci8ROp+v0TyADKyCIQEvg3cDFwi+/3nzBdeeHBER1PVEqA1bJQSJ/cdtdKi7OHHlMaUspLpJQTAvy818Zh+UDz2tYDgULf+MAA48HO/YyUcoaUckZqampXLqMFERdeSPzv/w8lJQUcDkRkJFF33E78z/tXDaSYb34DERkBzZ7YRWQk0Xfcjhriiqnuv89T+5e/IuvrkQ0NyIYG6p75F7VP/6OnxD7DeBt4Dmjw/biBtzAVSXgQQjDr679CdUYifE5vxe7AER3L5Fu/Hza5LPoWYXGEN51ciBW0dISPB17htCN8GTDS5wjfDNwPbMR0hP9FSvlxe+foTkd4I9IwMCqrUGKiEY7+WVzNe/gI1b/9He7161ESEoj5+r1E33VnyKafomkzME76J9aJhAQy9+7ubnHPQBYBgcJZo4EVBF5c9w7VhcfI+eRFqguOkjpmGqMu+xIR8VYTqbONYI7wcEVPXQv8BUgFKoEdUsrLfNseBb4KaMADjRFSQogZnA65/QS4P5SQ255QGhZQMDgb9MCmlMwTxxFW+ep2mIvpz2iNANYC/fNhxOLMoa9FT70DvBNk22+A3wQY3wJM6GHRLELENno02r59/uPDhlkKIyRGAnsDjGdiKQyLvoz17bboFAk//xkiomXLVRERQfwvfxEegfod3wNat6x1Aj8MgywWFqFjKQ2LTuGcdy7Jr72K49y5KCkpOGbPIvmlF4i46MJwi9ZPmIKZ2zoXSAGmAX8G5odRJguL9rEKFlp0Guc5M0h94/Vwi9G/KNoBn/8ICjZBdBrM/zFMeapFJJuFRV/GUhoWFr3FyT3wn3ngrTNfu6vh429DbRHMfyS8sllYhIhlnrKw6C1W/hK89S3HvHWw6jfgbQh8jIVFH8NSGhYWvUXBZgIWMhACqk74j1tY9EEspWFh0VskBSnRYmgQY5V4t+gfWErDwqK3OP+nYI9qOWaLhEl3QERceGSysOggltKwsOgtss+Ha1+A2CxQHaYCmX4vLPxbuCWzsAgZK3rKwqI3GXc9jL0OXJXgiAG1fzTusrBoxFIaFha9jRAQ2X/6r1hYNMcyT1lYWFhYhIylNCwsLCwsQsYyT1n0G1w1bvZ8eIC8rYVEJ0UyafE4MsYPCLdYFhZnFZbSsOgXuKrdvPXARzRUuzC8BiVA/o4i5tw9g3GXjQy3eBYWZw2WecqiX7Dr/f24fAqjEc2ts/4/W9HcWhgls7A4u7CUhkW/IG9LAXozhdGIUATleZW9L5CFxVmKpTQs+gWR8a0bFpkYmkFErLOXpbGwOHuxlIZFv2DS4rHYnGqLMaEKkrMTiUuPDZNUFhZnH5bSsOh23JqLbSe3suPUdryGt1vmHDQtkxm3TUZ1qDii7KgOlZRhSVz26PndMr+FhUVoWNFTFt3KhsL1/HHrkyhCQQIKgh/P+gkTUyd1ee7J14xj7IKRlB2rIDIhgoQsq8ifhUVvY600LLqN0oZSntzye1y6i3qtngatnjqtjl9t+CV1jd3quogjyk7G+AGWwrCwCBOW0rDoNlaeWI6Bf4QTwIbCdb0sjYWFRU8QFqUhhHhCCHFACLFLCPGOECKh2bZHhBCHhRA5QojLmo1PF0Ls9m37sxBChEN2i+DUemvRDP+cCV3q1Gn1AY6wsLDob4RrpfE5MEFKOQk4CDwCIIQYB9wCjAcuB/4uhGgMmXkauBcY6fu5vLeFtmib6WkziFD9Q2OFEEwdMDUMEllYWHQ3YXGESymXNHu5AbjB9/di4DUppRs4JoQ4DMwUQuQCcVLK9QBCiBeAa4BPek3ofoyUkrc3neDldblUN3iYNCiR+y8bzbABMd16nvHJE5iaNp3tJ7fi0l0ARKgRXDT4YgbFDu7Wc1lYWISHvhA99VXgdd/fWZhKpJF835jX93fr8YAIIe7FXJUweLB1s/rbkoO8tTkPly+jesPhUnbmVfDCN+cyMCmqnaNDRwjBQ+c8zIbC9Sw/8QU2xcYlQy5l2oDp3XYOCwuL8NJjSkMIsRRID7DpUSnle759HgU04OXGwwLsL9sYD4iU8hngGYAZM2YE3e9soKbByxub8vBopx3UEnB7dZ5ffZRHF0/o1vMpQmFu1rnMzTq3W+e1sLDoG/SY0pBSXtLWdiHEXcAi4GIpZeONPR8Y1Gy3gUChb3xggHGLdjhRXo9dVVooDQBdwt78qjBJZWFh0V8JV/TU5cBDwNVSyuZhNe8DtwghnEKIoZgO701SyiKgRggx2xc1dSfwXq8L3g9Ji4/Aqwco9CdgSHJ0GCSysLDoz4QreuqvQCzwuRBihxDiHwBSyr3AG8A+4FPgPiml7jvmm8CzwGHgCJYTPCSSY5ycN2YATlvLf7XTpnDXeUPDJJWFhUV/RZy2DJ2ZzJgxQ27ZsiXcYnQLRk0N9W//D+++fdjHjyfqumtRYtsv1uf26vzp0wN8tKMQw5CkxkXw4KKxzBmZ2gtSW1hY9EeEEFullDP8xi2l0T/Q8vIoWXQVRn0DNDRAVCRKZBSpH3+IbeDA9icANN3A5dWJdtqwciMtLCzaIpjSsMqI9BMqf/wTjIpKU2EA1DdgVFZS+ehPQ57DpirERNgthWFhYdFpLKXRD5BS4l61CoxWDm1dx71iRVhksrCwODuxlEZ/wRYkOjrYuIWFhUUPYCmNfoAQgsirrwKHveUGh4OoxYvDI5SFhcVZiaU0+gkJv/wFtpEjEdHREBmBiI7GPmoU8b/4WbhFs7CwOIuwbBv9BCU+ngGffYpn40a0Q4exjRyBY9Ysy6ltYWHRq1hKox8hhMA5ezbO2bPDLYqFhcVZimWesrCwsLAIGUtpWFhYWFiEjKU0LCwsLCxCxlIaFhYWFhYhYykNCwsLC4uQOeMLFgohSoDj4ZYjCClAabiF6Casa+mbWNfSN+kP1zJESulXCvuMVxp9GSHElkBVJPsj1rX0Taxr6Zv052uxzFMWFhYWFiFjKQ0LCwsLi5CxlEZ4eSbcAnQj1rX0Taxr6Zv022uxfBoWFhYWFiFjrTQsLCwsLELGUhoWFhYWFiFjKY0wIIS4XAiRI4Q4LIR4ONzydBYhxCAhxHIhxH4hxF4hxHfDLVNXEUKoQojtQogPwy1LVxBCJAgh3hJCHPD9f+aEW6bOIoT4nu/ztUcI8aoQIiLcMnUEIcR/hBCnhBB7mo0lCSE+F0Ic8v1ODKeMHcFSGr2MEEIF/gZcAYwDbhVCjAuvVJ1GA34gpRwLzAbu68fX0sh3gf3hFqIbeAr4VEo5BphMP70mIUQW8B1ghpRyAqACt4RXqg7zX+DyVmMPA8uklCOBZb7X/QJLafQ+M4HDUsqjUkoP8BrQL3u2SimLpJTbfH/XYN6YssIrVecRQgwEFgLPhluWriCEiAPOA/4NIKX0SCkrwypU17ABkUIIGxAFFIZZng4hpVwFlLcaXgw87/v7eeCa3pSpK1hKo/fJAk40e51PP77RNiKEyAamAhvDLEpX+BPwIGCEWY6uMgwoAZ7zmdqeFUJEh1uoziClLACeBPKAIqBKSrkkvFJ1C2lSyiIwH76AAWGWJ2QspdH7BOrP2q/jnoUQMcDbwANSyupwy9MZhBCLgFNSyq3hlqUbsAHTgKellFOBOvqR+aM5Plv/YmAokAlECyFuD69UZzeW0uh98oFBzV4PpJ8tt5sjhLBjKoyXpZT/C7c8XeBc4GohRC6myfAiIcRL4RWp0+QD+VLKxlXfW5hKpD9yCXBMSlkipfQC/wPmhlmm7uCkECIDwPf7VJjlCRlLafQ+m4GRQoihQggHplPv/TDL1CmEEALTbr5fSvnHcMvTFaSUj0gpB0opszH/J19IKfvlE62Ushj+f3t3DhpFGIZx/P9gBE9QiZ2IkMLCFMoqeDQeEDsJGCuRjYiFoJ02iggeoFjahzSieOKBELeJjSAxmnihiCIq2GmhYhN5LeZbXNc4zm6U9Xh+sLAzO7PfVyy8+83OPi+vJC1Mu9YBj1o4pYl4CSyXNC193tbxl/6oX+cyUE7Py8ClFs6lIW2tnsD/JiLGJO0EBsjuBOmLiIctnlazVgFbgPuSRtK+vRFxrXVTsmQXcDJ9MXkObG3xfJoSEbcknQPukN2td5e/LIJD0ilgNdAu6TVwADgKnJG0jawwbmrdDBvjGBEzMyvMl6fMzKwwFw0zMyvMRcPMzApz0TAzs8JcNMzMrDAXDbMckj5LGql5LGjiPbp/Z5CjpHJKS30qqfzzM8ya51tuzXJI+hARMyb4Hv3A1Yg418A5bRExVuC4OcBtYClZHM0wUIqId01O1yyXVxpmDZJUknRD0rCkgZo4iO2ShiSNSjqf/sW8EtgAHE8rlQ5Jg5KWpnPaU3QJknolnZV0BbguaXrqxTCUggfHS0NeD1Qi4m0qFBW+j+E2+2VcNMzyTa25NHUxZW2dAHoiogT0AUfSsRciYllEVPtXbIuIm2SREXsiYnFEPPvJeCuAckSsBfaRxZksA9aQFZ76tNp/MjXZ/lyOETHL9ykiFlc3JHUCnUAli0JiEllkN0CnpMPALGAGWVRMoyoRUe290EUWorg7bU8B5vNt9tI/l5psfzYXDbPGCHgYEeO1T+0HuiNiVFIvWd7QeMb4usqvb136sW6sjRHxJGc+r+vGmQcM5hxvNiG+PGXWmCfA3GrPbUmTJS1Kr80E3qRLWJtrznmfXqt6AZTS856csQaAXSndFUlLfnBMl6TZqfdEF82tcMwKcdEwa0Bq0dsDHJM0Cozwtb/DfrLOhRXgcc1pp4E96cfsDrJOdDsk3QTac4Y7BEwG7kl6kLbr5/M27R9Kj4M1l7fMfjnfcmtmZoV5pWFmZoW5aJiZWWEuGmZmVpiLhpmZFeaiYWZmhblomJlZYS4aZmZW2BebOipt60fcnwAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"fig, ax = plt.subplots()\n",
"ax.scatter(X2[::5, 0], y2[::5], c=np.arange(0, len(X2), 5) // 100, cmap=\"Set1\",\n",
" label=\"Partition\")\n",
"ax.set(xlabel=\"Feature 0\", ylabel=\"target\", title=\"Non-stationary data (by partition)\");"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's fit two estimators:\n",
"\n",
"1. One `BlockwiseVotingRegressor` on the entire dataset (which fits a `LinearRegression` on each partition)\n",
"2. One `LinearRegression` on a sample from the entire dataset"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:10.988822Z",
"iopub.status.busy": "2021-07-26T20:19:10.988143Z",
"iopub.status.idle": "2021-07-26T20:19:11.087075Z",
"shell.execute_reply": "2021-07-26T20:19:11.086600Z"
}
},
"outputs": [],
"source": [
"subestimator = sklearn.linear_model.LinearRegression()\n",
"clf = dask_ml.ensemble.BlockwiseVotingRegressor(\n",
" subestimator,\n",
")\n",
"clf.fit(X2, y2)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:11.095118Z",
"iopub.status.busy": "2021-07-26T20:19:11.090747Z",
"iopub.status.idle": "2021-07-26T20:19:11.151839Z",
"shell.execute_reply": "2021-07-26T20:19:11.152220Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"LinearRegression()"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X_sampled, y_sampled = dask.compute(X2[::10], y2[::10])\n",
"\n",
"subestimator.fit(X_sampled, y_sampled)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Comparing the scores, we find that the sampled dataset performs much better, despite training on less data."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:11.183493Z",
"iopub.status.busy": "2021-07-26T20:19:11.159756Z",
"iopub.status.idle": "2021-07-26T20:19:11.366938Z",
"shell.execute_reply": "2021-07-26T20:19:11.366071Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"-11.154062558523927"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.score(X2, y2)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"execution": {
"iopub.execute_input": "2021-07-26T20:19:11.371864Z",
"iopub.status.busy": "2021-07-26T20:19:11.371417Z",
"iopub.status.idle": "2021-07-26T20:19:11.453736Z",
"shell.execute_reply": "2021-07-26T20:19:11.454129Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.03517659566372422"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"subestimator.score(X2, y2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This shows that ensuring your needs to be relatively uniform across partitions. Even including the standard controls to normalize whatever underlying force is generating the non-stationary data (e.g. a time trend compontent or differencing timeseries data, dummy variables for geographic regions, etc) is not sufficient when your dataset is partioned by the non-uniform variable. You would still need to either shuffle your data prior to fitting, or just sample and fit the sub-estimator on the sub-sample that fits in memory."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}