{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Automate Machine Learning with TPOT\n", "===================================\n", "\n", "This example shows how [TPOT](https://epistasislab.github.io/tpot/) can be used with Dask.\n", "\n", "TPOT is an [automated machine learning](https://en.wikipedia.org/wiki/Automated_machine_learning) library.\n", "It evaluates many scikit-learn pipelines and hyperparameter combinations to find a model that works well for your data. Evaluating all these computations is computationally expensive, but ammenable to parallelism. TPOT can use Dask to distribute these computations on a cluster of machines.\n", "\n", "This notebook can be run interactively on the [dask examples binder](https://github.com/dask/dask-examples).\n", "The following video shows a larger version of this notebook on a cluster." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2021-10-14T16:51:03.563854Z", "iopub.status.busy": "2021-10-14T16:51:03.563172Z", "iopub.status.idle": "2021-10-14T16:51:03.570901Z", "shell.execute_reply": "2021-10-14T16:51:03.571672Z" } }, "outputs": [ { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from IPython.display import HTML\n", "\n", "HTML('
')" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2021-10-14T16:51:03.575267Z", "iopub.status.busy": "2021-10-14T16:51:03.574063Z", "iopub.status.idle": "2021-10-14T16:51:08.119174Z", "shell.execute_reply": "2021-10-14T16:51:08.119594Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting tpot\r\n", " Downloading TPOT-0.11.7-py3-none-any.whl (87 kB)\r\n", "\u001b[?25l\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " |███▊ | 10 kB 5.1 MB/s eta 0:00:01\r", " |███████▌ | 20 kB 8.5 MB/s eta 0:00:01\r", " |███████████▎ | 30 kB 11.2 MB/s eta 0:00:01\r", " |███████████████ | 40 kB 13.3 MB/s eta 0:00:01\r", " |██████████████████▉ | 51 kB 13.4 MB/s eta 0:00:01\r", " |██████████████████████▋ | 61 kB 13.6 MB/s eta 0:00:01\r", " |██████████████████████████▎ | 71 kB 14.4 MB/s eta 0:00:01\r", " |██████████████████████████████ | 81 kB 15.6 MB/s eta 0:00:01\r", " |████████████████████████████████| 87 kB 7.9 MB/s \r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\u001b[?25hCollecting deap>=1.2\r\n", " Downloading deap-1.3.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (160 kB)\r\n", "\u001b[?25l\r", "\r", " |██ | 10 kB 19.3 MB/s eta 0:00:01\r", " |████ | 20 kB 25.1 MB/s eta 0:00:01\r", " |██████▏ | 30 kB 27.0 MB/s eta 0:00:01\r", " |████████▏ | 40 kB 28.0 MB/s eta 0:00:01\r", " |██████████▏ | 51 kB 29.5 MB/s eta 0:00:01\r", " |████████████▎ | 61 kB 30.7 MB/s eta 0:00:01\r", " |██████████████▎ | 71 kB 31.2 MB/s eta 0:00:01\r", " |████████████████▍ | 81 kB 31.6 MB/s eta 0:00:01\r", " |██████████████████▍ | 92 kB 30.7 MB/s eta 0:00:01\r", " |████████████████████▍ | 102 kB 31.0 MB/s eta 0:00:01\r", " |██████████████████████▌ | 112 kB 31.0 MB/s eta 0:00:01\r", " |████████████████████████▌ | 122 kB 31.0 MB/s eta 0:00:01\r", " |██████████████████████████▋ | 133 kB 31.0 MB/s eta 0:00:01\r", " |████████████████████████████▋ | 143 kB 31.0 MB/s eta 0:00:01\r", " |██████████████████████████████▋ | 153 kB 31.0 MB/s eta 0:00:01\r", " |████████████████████████████████| 160 kB 31.0 MB/s \r\n", "\u001b[?25hRequirement already satisfied: numpy>=1.16.3 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from tpot) (1.21.2)\r\n", "Requirement already satisfied: tqdm>=4.36.1 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from tpot) (4.62.3)\r\n", "Requirement already satisfied: joblib>=0.13.2 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from tpot) (1.1.0)\r\n", "Requirement already satisfied: xgboost>=1.1.0 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from tpot) (1.4.2)\r\n", "Collecting stopit>=1.1.1\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Downloading stopit-1.1.2.tar.gz (18 kB)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Preparing metadata (setup.py) ... \u001b[?25l-\b \bdone" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "\u001b[?25hCollecting update-checker>=0.16\r\n", " Downloading update_checker-0.18.0-py3-none-any.whl (7.0 kB)\r\n", "Requirement already satisfied: scikit-learn>=0.22.0 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from tpot) (0.23.2)\r\n", "Requirement already satisfied: pandas>=0.24.2 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from tpot) (1.3.3)\r\n", "Requirement already satisfied: scipy>=1.3.1 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from tpot) (1.7.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from pandas>=0.24.2->tpot) (2.8.2)\r\n", "Requirement already satisfied: pytz>=2017.3 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from pandas>=0.24.2->tpot) (2021.3)\r\n", "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from scikit-learn>=0.22.0->tpot) (3.0.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: requests>=2.3.0 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from update-checker>=0.16->tpot) (2.25.1)\r\n", "Requirement already satisfied: six>=1.5 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas>=0.24.2->tpot) (1.16.0)\r\n", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from requests>=2.3.0->update-checker>=0.16->tpot) (1.26.7)\r\n", "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from requests>=2.3.0->update-checker>=0.16->tpot) (4.0.0)\r\n", "Requirement already satisfied: idna<3,>=2.5 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from requests>=2.3.0->update-checker>=0.16->tpot) (2.10)\r\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages (from requests>=2.3.0->update-checker>=0.16->tpot) (2021.10.8)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Building wheels for collected packages: stopit\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Building wheel for stopit (setup.py) ... \u001b[?25l-" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b \b\\" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b \bdone\r\n", "\u001b[?25h Created wheel for stopit: filename=stopit-1.1.2-py3-none-any.whl size=11952 sha256=2caf8e163d84efa030d87b69b668de73c9f29571bc88e7b1a7a855346fe67fd7\r\n", " Stored in directory: /home/runner/.cache/pip/wheels/a8/bb/8f/6b9328d23c2dcedbfeb8498b9f650d55d463089e3b8fc0bfb2\r\n", "Successfully built stopit\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: update-checker, stopit, deap, tpot\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed deap-1.3.1 stopit-1.1.2 tpot-0.11.7 update-checker-0.18.0\r\n" ] } ], "source": [ "!pip install tpot" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2021-10-14T16:51:08.124190Z", "iopub.status.busy": "2021-10-14T16:51:08.123675Z", "iopub.status.idle": "2021-10-14T16:51:09.553021Z", "shell.execute_reply": "2021-10-14T16:51:09.554055Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages/tpot/builtins/__init__.py:36: UserWarning: Warning: optional dependency `torch` is not available. - skipping import of NN models.\n", " warnings.warn(\"Warning: optional dependency `torch` is not available. - skipping import of NN models.\")\n" ] } ], "source": [ "import tpot\n", "from tpot import TPOTClassifier\n", "from sklearn.datasets import load_digits\n", "from sklearn.model_selection import train_test_split" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup Dask\n", "\n", "We first start a Dask client in order to get access to the Dask dashboard, which will provide progress and performance metrics. \n", "\n", "You can view the dashboard by clicking on the dashboard link after you run the cell." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2021-10-14T16:51:09.557790Z", "iopub.status.busy": "2021-10-14T16:51:09.556687Z", "iopub.status.idle": "2021-10-14T16:51:12.397639Z", "shell.execute_reply": "2021-10-14T16:51:12.398257Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages/distributed/node.py:151: UserWarning: Port 8787 is already in use.\n", "Perhaps you already have a cluster running?\n", "Hosting the HTTP server on port 33083 instead\n", " warnings.warn(\n" ] }, { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "

Client

\n", "\n", "		\n", "

Cluster

\n", "
    \n", "
  • Workers: 4
    • \n", "
  • Cores: 4
    • \n", "
  • Memory: 7.29 GB
    • \n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from dask.distributed import Client\n", "client = Client(n_workers=4, threads_per_worker=1)\n", "client" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create Data\n", "\n", "We'll use the digits dataset.\n", "To ensure the example runs quickly, we'll make the training dataset relatively small." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2021-10-14T16:51:12.401650Z", "iopub.status.busy": "2021-10-14T16:51:12.401112Z", "iopub.status.idle": "2021-10-14T16:51:12.491750Z", "shell.execute_reply": "2021-10-14T16:51:12.492586Z" } }, "outputs": [], "source": [ "digits = load_digits()\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " digits.data,\n", " digits.target,\n", " train_size=0.05,\n", " test_size=0.95,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These are just small, in-memory NumPy arrays. This example is not applicable to larger-than-memory Dask arrays." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using Dask\n", "\n", "TPOT follows the scikit-learn API; we specify a `TPOTClassifier` with a few hyperparameters, and then fit it on some data.\n", "By default, TPOT trains on your single machine.\n", "To ensure your cluster is used, specify the `use_dask` keyword." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2021-10-14T16:51:12.496907Z", "iopub.status.busy": "2021-10-14T16:51:12.494989Z", "iopub.status.idle": "2021-10-14T16:51:12.500706Z", "shell.execute_reply": "2021-10-14T16:51:12.501665Z" } }, "outputs": [], "source": [ "# scale up: Increase the TPOT parameters like population_size, generations\n", "tp = TPOTClassifier(\n", " generations=2,\n", " population_size=10,\n", " cv=2,\n", " n_jobs=-1,\n", " random_state=0,\n", " verbosity=0,\n", " config_dict=tpot.config.classifier_config_dict_light,\n", " use_dask=True,\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2021-10-14T16:51:12.505346Z", "iopub.status.busy": "2021-10-14T16:51:12.504171Z", "iopub.status.idle": "2021-10-14T16:51:19.967589Z", "shell.execute_reply": "2021-10-14T16:51:19.968427Z" } }, "outputs": [ { "data": { "text/plain": [ "TPOTClassifier(config_dict={'sklearn.cluster.FeatureAgglomeration': {'affinity': ['euclidean',\n", " 'l1',\n", " 'l2',\n", " 'manhattan',\n", " 'cosine'],\n", " 'linkage': ['ward',\n", " 'complete',\n", " 'average']},\n", " 'sklearn.decomposition.PCA': {'iterated_power': range(1, 11),\n", " 'svd_solver': ['randomized']},\n", " 'sklearn.feature_selection.SelectFwe': {'alpha': array([0. , 0.001, 0.002, 0.003, 0.004, 0.005, 0.006, 0.007...\n", " 'max']},\n", " 'sklearn.preprocessing.RobustScaler': {},\n", " 'sklearn.preprocessing.StandardScaler': {},\n", " 'sklearn.tree.DecisionTreeClassifier': {'criterion': ['gini',\n", " 'entropy'],\n", " 'max_depth': range(1, 11),\n", " 'min_samples_leaf': range(1, 21),\n", " 'min_samples_split': range(2, 21)},\n", " 'tpot.builtins.ZeroCount': {}},\n", " cv=2, generations=2, n_jobs=-1, population_size=10,\n", " random_state=0, use_dask=True)" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tp.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Learn More\n", "\n", "See the [Dask-ML](http://ml.dask.org/) and [TPOT](https://epistasislab.github.io/tpot/) documenation for more information on using Dask and TPOT." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" } }, "nbformat": 4, "nbformat_minor": 2 }