![](\"images/scikit-learn-logo-notext.png\"/){kind=logo}
![](\"images/joblib_logo.svg\"){kind=logo}
\n",
"\n",
"Scikit-learn uses [joblib](http://joblib.readthedocs.io/) for single-machine parallelism. This lets you train most estimators (anything that accepts an `n_jobs` parameter) using all the cores of your laptop or workstation.\n",
"\n",
"Alternatively, Scikit-Learn can use Dask for parallelism. This lets you train those estimators using all the cores of your *cluster* without significantly changing your code.\n",
"\n",
"This is most useful for training large models on medium-sized datasets. You may have a large model when searching over many hyper-parameters, or when using an ensemble method with many individual estimators. For too small datasets, training times will typically be small enough that cluster-wide parallelism isn't helpful. For too large datasets (larger than a single machine's memory), the scikit-learn estimators may not be able to cope (see below)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create Scikit-Learn Estimator"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:03.344174Z",
"iopub.status.busy": "2021-01-14T10:43:03.343114Z",
"iopub.status.idle": "2021-01-14T10:43:03.603088Z",
"shell.execute_reply": "2021-01-14T10:43:03.603456Z"
}
},
"outputs": [],
"source": [
"from sklearn.datasets import make_classification\n",
"from sklearn.svm import SVC\n",
"from sklearn.model_selection import GridSearchCV\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll use scikit-learn to create a pair of small random arrays, one for the features `X`, and one for the target `y`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:03.607354Z",
"iopub.status.busy": "2021-01-14T10:43:03.606777Z",
"iopub.status.idle": "2021-01-14T10:43:03.615666Z",
"shell.execute_reply": "2021-01-14T10:43:03.615983Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([[-1.06377997, 0.67640868, 1.06935647, -0.21758002, 0.46021477,\n",
" -0.39916689, -0.07918751, 1.20938491, -0.78531472, -0.17218611,\n",
" -1.08535744, -0.99311895, 0.30693511, 0.06405769, -1.0542328 ,\n",
" -0.52749607, -0.0741832 , -0.35562842, 1.05721416, -0.90259159],\n",
" [ 0.0708476 , -1.69528125, 2.44944917, -0.5304942 , -0.93296221,\n",
" 2.86520354, 2.43572851, -1.61850016, 1.30071691, 0.34840246,\n",
" 0.54493439, 0.22532411, 0.60556322, -0.19210097, -0.06802699,\n",
" 0.9716812 , -1.79204799, 0.01708348, -0.37566904, -0.62323644],\n",
" [ 0.94028404, -0.49214582, 0.67795602, -0.22775445, 1.40175261,\n",
" 1.23165333, -0.77746425, 0.01561602, 1.33171299, 1.08477266,\n",
" -0.97805157, -0.05012039, 0.94838552, -0.17342825, -0.47767184,\n",
" 0.76089649, 1.00115812, -0.06946407, 1.35904607, -1.18958963],\n",
" [-0.29951677, 0.75988955, 0.18280267, -1.55023271, 0.33821802,\n",
" 0.36324148, -2.10052547, -0.4380675 , -0.16639343, -0.34083531,\n",
" 0.42435643, 1.17872434, 2.8314804 , 0.14241375, -0.20281911,\n",
" 2.40571546, 0.31330473, 0.40435568, -0.28754632, -2.8478034 ],\n",
" [-2.63062675, 0.23103376, 0.04246253, 0.47885055, 1.54674163,\n",
" 1.6379556 , -1.53207229, -0.73444479, 0.46585484, 0.4738362 ,\n",
" 0.98981401, -1.06119392, -0.88887952, 1.23840892, -0.57282854,\n",
" -1.27533949, 1.0030065 , -0.47712843, 0.09853558, 0.52780407]])"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X, y = make_classification(n_samples=1000, random_state=0)\n",
"X[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll fit a [Support Vector Classifier](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html), using [grid search](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) to find the best value of the $C$ hyperparameter."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:03.620040Z",
"iopub.status.busy": "2021-01-14T10:43:03.619626Z",
"iopub.status.idle": "2021-01-14T10:43:03.622864Z",
"shell.execute_reply": "2021-01-14T10:43:03.622474Z"
}
},
"outputs": [],
"source": [
"param_grid = {\"C\": [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0],\n",
" \"kernel\": ['rbf', 'poly', 'sigmoid'],\n",
" \"shrinking\": [True, False]}\n",
"\n",
"grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),\n",
" param_grid=param_grid,\n",
" return_train_score=False,\n",
" iid=True,\n",
" cv=3,\n",
" n_jobs=-1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To fit that normally, we would call\n",
"\n",
"```python\n",
"grid_search.fit(X, y)\n",
"```\n",
"\n",
"To fit it using the cluster, we just need to use a context manager provided by joblib."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:03.625754Z",
"iopub.status.busy": "2021-01-14T10:43:03.625285Z",
"iopub.status.idle": "2021-01-14T10:43:10.808415Z",
"shell.execute_reply": "2021-01-14T10:43:10.808998Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/share/miniconda3/envs/dask-examples/lib/python3.8/site-packages/sklearn/model_selection/_search.py:847: FutureWarning: The parameter 'iid' is deprecated in 0.22 and will be removed in 0.24.\n",
" warnings.warn(\n"
]
}
],
"source": [
"import joblib\n",
"\n",
"with joblib.parallel_backend('dask'):\n",
" grid_search.fit(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We fit 48 different models, one for each hyper-parameter combination in `param_grid`, distributed across the cluster. At this point, we have a regular scikit-learn model, which can be used for prediction, scoring, etc."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:10.824054Z",
"iopub.status.busy": "2021-01-14T10:43:10.815486Z",
"iopub.status.idle": "2021-01-14T10:43:10.835113Z",
"shell.execute_reply": "2021-01-14T10:43:10.834403Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" mean_fit_time std_fit_time mean_score_time std_score_time param_C \\\n",
"0 0.281757 0.028381 0.031847 0.006431 0.001 \n",
"1 0.302007 0.040334 0.023815 0.002342 0.001 \n",
"2 0.148769 0.010645 0.015993 0.003539 0.001 \n",
"3 0.170737 0.020296 0.012334 0.001556 0.001 \n",
"4 0.331724 0.007535 0.032123 0.004971 0.001 \n",
"\n",
" param_kernel param_shrinking \\\n",
"0 rbf True \n",
"1 rbf False \n",
"2 poly True \n",
"3 poly False \n",
"4 sigmoid True \n",
"\n",
" params split0_test_score \\\n",
"0 {'C': 0.001, 'kernel': 'rbf', 'shrinking': True} 0.502994 \n",
"1 {'C': 0.001, 'kernel': 'rbf', 'shrinking': False} 0.502994 \n",
"2 {'C': 0.001, 'kernel': 'poly', 'shrinking': True} 0.502994 \n",
"3 {'C': 0.001, 'kernel': 'poly', 'shrinking': Fa... 0.502994 \n",
"4 {'C': 0.001, 'kernel': 'sigmoid', 'shrinking':... 0.502994 \n",
"\n",
" split1_test_score split2_test_score mean_test_score std_test_score \\\n",
"0 0.501502 0.501502 0.502 0.000704 \n",
"1 0.501502 0.501502 0.502 0.000704 \n",
"2 0.501502 0.501502 0.502 0.000704 \n",
"3 0.501502 0.501502 0.502 0.000704 \n",
"4 0.501502 0.501502 0.502 0.000704 \n",
"\n",
" rank_test_score \n",
"0 41 \n",
"1 41 \n",
"2 41 \n",
"3 41 \n",
"4 41 "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.DataFrame(grid_search.cv_results_).head()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:10.837747Z",
"iopub.status.busy": "2021-01-14T10:43:10.837223Z",
"iopub.status.idle": "2021-01-14T10:43:10.860477Z",
"shell.execute_reply": "2021-01-14T10:43:10.860794Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([0, 1, 1, 1, 0])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"grid_search.predict(X)[:5]"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:10.862699Z",
"iopub.status.busy": "2021-01-14T10:43:10.862302Z",
"iopub.status.idle": "2021-01-14T10:43:10.885638Z",
"shell.execute_reply": "2021-01-14T10:43:10.886078Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.972"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"grid_search.score(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For more on training scikit-learn models with distributed joblib, see the [dask-ml documentation](http://dask-ml.readthedocs.io/en/latest/joblib.html)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training on Large Datasets\n",
"\n",
"Most estimators in scikit-learn are designed to work on in-memory arrays. Training with larger datasets may require different algorithms.\n",
"\n",
"All of the algorithms implemented in Dask-ML work well on larger than memory datasets, which you might store in a [dask array](http://dask.pydata.org/en/latest/array.html) or [dataframe](http://dask.pydata.org/en/latest/dataframe.html)."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:10.888552Z",
"iopub.status.busy": "2021-01-14T10:43:10.887870Z",
"iopub.status.idle": "2021-01-14T10:43:11.043196Z",
"shell.execute_reply": "2021-01-14T10:43:11.042760Z"
}
},
"outputs": [],
"source": [
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:11.046690Z",
"iopub.status.busy": "2021-01-14T10:43:11.045916Z",
"iopub.status.idle": "2021-01-14T10:43:11.504312Z",
"shell.execute_reply": "2021-01-14T10:43:11.503305Z"
}
},
"outputs": [],
"source": [
"import dask_ml.datasets\n",
"import dask_ml.cluster\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, we'll use `dask_ml.datasets.make_blobs` to generate some random *dask* arrays."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-14T10:43:11.511298Z",
"iopub.status.busy": "2021-01-14T10:43:11.510888Z",
"iopub.status.idle": "2021-01-14T10:43:11.871844Z",
"shell.execute_reply": "2021-01-14T10:43:11.872241Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | mean_fit_time | \n", "std_fit_time | \n", "mean_score_time | \n", "std_score_time | \n", "param_C | \n", "param_kernel | \n", "param_shrinking | \n", "params | \n", "split0_test_score | \n", "split1_test_score | \n", "split2_test_score | \n", "mean_test_score | \n", "std_test_score | \n", "rank_test_score | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "0.281757 | \n", "0.028381 | \n", "0.031847 | \n", "0.006431 | \n", "0.001 | \n", "rbf | \n", "True | \n", "{'C': 0.001, 'kernel': 'rbf', 'shrinking': True} | \n", "0.502994 | \n", "0.501502 | \n", "0.501502 | \n", "0.502 | \n", "0.000704 | \n", "41 | \n", "
| 1 | \n", "0.302007 | \n", "0.040334 | \n", "0.023815 | \n", "0.002342 | \n", "0.001 | \n", "rbf | \n", "False | \n", "{'C': 0.001, 'kernel': 'rbf', 'shrinking': False} | \n", "0.502994 | \n", "0.501502 | \n", "0.501502 | \n", "0.502 | \n", "0.000704 | \n", "41 | \n", "
| 2 | \n", "0.148769 | \n", "0.010645 | \n", "0.015993 | \n", "0.003539 | \n", "0.001 | \n", "poly | \n", "True | \n", "{'C': 0.001, 'kernel': 'poly', 'shrinking': True} | \n", "0.502994 | \n", "0.501502 | \n", "0.501502 | \n", "0.502 | \n", "0.000704 | \n", "41 | \n", "
| 3 | \n", "0.170737 | \n", "0.020296 | \n", "0.012334 | \n", "0.001556 | \n", "0.001 | \n", "poly | \n", "False | \n", "{'C': 0.001, 'kernel': 'poly', 'shrinking': Fa... | \n", "0.502994 | \n", "0.501502 | \n", "0.501502 | \n", "0.502 | \n", "0.000704 | \n", "41 | \n", "
| 4 | \n", "0.331724 | \n", "0.007535 | \n", "0.032123 | \n", "0.004971 | \n", "0.001 | \n", "sigmoid | \n", "True | \n", "{'C': 0.001, 'kernel': 'sigmoid', 'shrinking':... | \n", "0.502994 | \n", "0.501502 | \n", "0.501502 | \n", "0.502 | \n", "0.000704 | \n", "41 | \n", "
\n",
"
| \n",
"\n", "\n", " | \n", "