Live Notebook

You can run this notebook in a live session Binder or view it on Github.

Generalized Linear Models

This notebook introduces the algorithms within Dask-GLM for Generalized Linear Models.

Start Dask Client for Dashboard

Starting the Dask Client is optional. It will provide a dashboard which is useful to gain insight on the computation.

The link to the dashboard will become visible when you create the client below. We recommend having it open on one side of your screen while using your notebook on the other side. This can take some effort to arrange your windows, but seeing them both at the same is very useful when learning.

[1]:
from dask.distributed import Client, progress
client = Client(processes=False, threads_per_worker=4,
                n_workers=1, memory_limit='2GB')
client
[1]:

Client

Cluster

  • Workers: 1
  • Cores: 4
  • Memory: 2.00 GB

Make a random dataset

[2]:
from dask_glm.datasets import make_regression
X, y = make_regression(n_samples=200000, n_features=100, n_informative=5, chunksize=10000)
X
[2]:
Array Chunk
Bytes 160.00 MB 8.00 MB
Shape (200000, 100) (10000, 100)
Count 20 Tasks 20 Chunks
Type float64 numpy.ndarray
100 200000
[3]:
import dask
X, y = dask.persist(X, y)

Solve with a GLM algorithm

We also recommend looking at the “Graph” dashboard during execution if available

[4]:
import dask_glm.algorithms

b = dask_glm.algorithms.admm(X, y, max_iter=5)

Solve with a difference GLM algorithm

[5]:
b = dask_glm.algorithms.proximal_grad(X, y, max_iter=5)
/home/travis/miniconda/envs/test/lib/python3.7/site-packages/dask/core.py:119: RuntimeWarning: overflow encountered in exp
  return func(*args2)

Customizable with different families and regularizers

The Dask-GLM project is nicely modular, allowing for different GLM families and regularizers, including a relatively straightforward interface for implementing custom ones.

[6]:
import dask_glm.families
import dask_glm.regularizers

family = dask_glm.families.Poisson()
regularizer = dask_glm.regularizers.ElasticNet()

b = dask_glm.algorithms.proximal_grad(
    X, y,
    max_iter=5,
    family=family,
    regularizer=regularizer,
)
/home/travis/miniconda/envs/test/lib/python3.7/site-packages/dask/core.py:119: RuntimeWarning: overflow encountered in exp
  return func(*args2)
/home/travis/miniconda/envs/test/lib/python3.7/site-packages/dask/core.py:119: RuntimeWarning: overflow encountered in exp
  return func(*args2)
[7]:
dask_glm.families.Poisson??
[8]:
dask_glm.regularizers.ElasticNet??
[ ]: