{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "Asynchronous Computation: Web Servers + Dask\n", "===========================\n", "\n", "Lets imagine a simple web server that serves both fast-loading pages and also performs some computation on slower loading pages. In our case this will be a simple Fibonnaci serving application, but you can imagine replacing the `fib` function for running a machine learning model on some input data, fetching results from a database, etc.." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:51:46.028040Z", "iopub.status.busy": "2022-05-16T13:51:46.027569Z", "iopub.status.idle": "2022-05-16T13:51:46.071102Z", "shell.execute_reply": "2022-05-16T13:51:46.070333Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import tornado.ioloop\n", "import tornado.web\n", "\n", "def fib(n):\n", " if n < 2:\n", " return n\n", " else:\n", " return fib(n - 1) + fib(n - 2)\n", " \n", "\n", "class FibHandler(tornado.web.RequestHandler):\n", " def get(self, n):\n", " result = fib(int(n))\n", " self.write(str(result))\n", "\n", " \n", "class FastHandler(tornado.web.RequestHandler):\n", " def get(self):\n", " self.write(\"Hello!\")\n", " \n", " \n", "def make_app():\n", " return tornado.web.Application([\n", " (r\"/fast\", FastHandler),\n", " (r\"/fib/(\\d+)\", FibHandler),\n", " ])\n", "\n", "\n", "app = make_app()\n", "app.listen(8000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Speed\n", "\n", "We know that users associate fast response time to authoritative content and trust, so we want to measure how fast our pages load. We're particularly interested in doing this during many simultaneous loads, simulating how our web server will respond when serving many users" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:51:46.075972Z", "iopub.status.busy": "2022-05-16T13:51:46.075783Z", "iopub.status.idle": "2022-05-16T13:51:46.084293Z", "shell.execute_reply": "2022-05-16T13:51:46.083484Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "import asyncio\n", "import tornado.httpclient\n", "\n", "client = tornado.httpclient.AsyncHTTPClient()\n", "\n", "from time import time\n", "\n", "async def measure(url, n=100):\n", " \"\"\" Get url n times concurrently. Print duration. \"\"\"\n", " start = time()\n", " futures = [client.fetch(url) for i in range(n)]\n", " results = await asyncio.gather(*futures)\n", " end = time()\n", " print(url, ', %d simultaneous requests, ' % n, 'total time: ', (end - start))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Timings\n", "\n", "We see that \n", "\n", "1. Tornado has about a 3-5ms roundtrip time\n", "2. It can run 100 such queries in around 100ms, so there is some nice concurrency happening\n", "3. Calling fib takes a while\n", "4. Calling fib 100 times takes around 100 times as long, there is not as much parallelism" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:51:46.087920Z", "iopub.status.busy": "2022-05-16T13:51:46.087741Z", "iopub.status.idle": "2022-05-16T13:51:46.097026Z", "shell.execute_reply": "2022-05-16T13:51:46.096390Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:8000/fast , 1 simultaneous requests, total time: 0.00574803352355957\n" ] } ], "source": [ "await measure('http://localhost:8000/fast', n=1)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:51:46.099595Z", "iopub.status.busy": "2022-05-16T13:51:46.099202Z", "iopub.status.idle": "2022-05-16T13:51:46.233753Z", "shell.execute_reply": "2022-05-16T13:51:46.233128Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:8000/fast , 100 simultaneous requests, total time: 0.12964129447937012\n" ] } ], "source": [ "await measure('http://localhost:8000/fast', n=100)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:51:46.236780Z", "iopub.status.busy": "2022-05-16T13:51:46.236246Z", "iopub.status.idle": "2022-05-16T13:51:46.350018Z", "shell.execute_reply": "2022-05-16T13:51:46.349189Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:8000/fib/28 , 1 simultaneous requests, total time: 0.1089627742767334\n" ] } ], "source": [ "await measure('http://localhost:8000/fib/28', n=1)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:51:46.353059Z", "iopub.status.busy": "2022-05-16T13:51:46.352742Z", "iopub.status.idle": "2022-05-16T13:51:57.026038Z", "shell.execute_reply": "2022-05-16T13:51:57.025033Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:8000/fib/28 , 100 simultaneous requests, total time: 10.668456077575684\n" ] } ], "source": [ "await measure('http://localhost:8000/fib/28', n=100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Blocking async\n", "\n", "In the example below we see that one call to the slow `fib/` route will unfortunately block other much faster requests:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:51:57.029702Z", "iopub.status.busy": "2022-05-16T13:51:57.029268Z", "iopub.status.idle": "2022-05-16T13:52:00.052428Z", "shell.execute_reply": "2022-05-16T13:52:00.051521Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:8000/fast , 1 simultaneous requests, total time: 3.017343759536743\n", "http://localhost:8000/fib/35 , 1 simultaneous requests, total time: 3.0177764892578125\n" ] } ], "source": [ "a = asyncio.ensure_future(measure('http://localhost:8000/fib/35', n=1))\n", "b = asyncio.ensure_future(measure('http://localhost:8000/fast', n=1))\n", "await b" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Discussion\n", "\n", "There are two problems/opportunities here:\n", "\n", "1. All of our `fib` calls are independent, we would like to run these computations in parallel with multiple cores or a nearby cluster.\n", "2. Our slow computationally intense `fib` requests can get in the way of our fast requests. One slow user can affect everyone else." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Asynchronous off-process computation with Dask\n", "\n", "To resolve both of these problems we will offload computation to other processes or computers using Dask. Because Dask is an async framework it can integrate nicely with Tornado or Asyncio." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:52:00.055560Z", "iopub.status.busy": "2022-05-16T13:52:00.055247Z", "iopub.status.idle": "2022-05-16T13:52:02.139460Z", "shell.execute_reply": "2022-05-16T13:52:02.138800Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Client

\n", "

Client-5be7eed7-d51f-11ec-9e43-000d3aeabb7a

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
Connection method: Cluster objectCluster type: distributed.LocalCluster
\n", " Dashboard: http://127.0.0.1:8787/status\n", "
\n", "\n", " \n", "
\n", "

Cluster Info

\n", "
\n", "
\n", "
\n", "
\n", "

LocalCluster

\n", "

e31271fd

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Workers: 2\n", "
\n", " Total threads: 2\n", " \n", " Total memory: 6.78 GiB\n", "
Status: runningUsing processes: True
\n", "\n", "
\n", " \n", "

Scheduler Info

\n", " \n", "\n", "
\n", "
\n", "
\n", "
\n", "

Scheduler

\n", "

Scheduler-cfdaae31-e429-445e-89a2-a9a7a73b3031

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Comm: tcp://127.0.0.1:44991\n", " \n", " Workers: 2\n", "
\n", " Dashboard: http://127.0.0.1:8787/status\n", " \n", " Total threads: 2\n", "
\n", " Started: Just now\n", " \n", " Total memory: 6.78 GiB\n", "
\n", "
\n", "
\n", "\n", "
\n", " \n", "

Workers

\n", " \n", "\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 0

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:43503\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: http://127.0.0.1:36375/status\n", " \n", " Memory: 3.39 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:38565\n", "
\n", " Local directory: /home/runner/work/dask-examples/dask-examples/applications/dask-worker-space/worker-uvdip27x\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 1

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: tcp://127.0.0.1:36349\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: http://127.0.0.1:46009/status\n", " \n", " Memory: 3.39 GiB\n", "
\n", " Nanny: tcp://127.0.0.1:38103\n", "
\n", " Local directory: /home/runner/work/dask-examples/dask-examples/applications/dask-worker-space/worker-r8ik19z6\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from dask.distributed import Client\n", "\n", "dask_client = await Client(asynchronous=True) # use local processes for now\n", "dask_client" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:52:02.143370Z", "iopub.status.busy": "2022-05-16T13:52:02.142870Z", "iopub.status.idle": "2022-05-16T13:52:02.153332Z", "shell.execute_reply": "2022-05-16T13:52:02.152689Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "def fib(n):\n", " if n < 2:\n", " return n\n", " else:\n", " return fib(n - 1) + fib(n - 2)\n", "\n", " \n", "class FibHandler(tornado.web.RequestHandler):\n", " async def get(self, n):\n", " future = dask_client.submit(fib, int(n)) # submit work to happen elsewhere\n", " result = await future\n", " self.write(str(result))\n", "\n", " \n", "class MainHandler(tornado.web.RequestHandler):\n", " async def get(self):\n", " self.write(\"Hello, world\")\n", "\n", " \n", "def make_app():\n", " return tornado.web.Application([\n", " (r\"/fast\", MainHandler),\n", " (r\"/fib/(\\d+)\", FibHandler),\n", " \n", " ])\n", "\n", "app = make_app()\n", "app.listen(9000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Performance changes\n", "\n", "By offloading the fib computation to Dask we acheive two things:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Parallel computing\n", "\n", "We can now support more requests in less time. The following experiment asks for `fib(28)` simultaneously from 20 requests. In the old version we computed these sequentially over a few seconds (the last person to request waited for a few seconds while their browser finished). In the new one many of these may be computed in parallel and so everyone gets an answer in a few hundred milliseconds." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:52:02.156060Z", "iopub.status.busy": "2022-05-16T13:52:02.155646Z", "iopub.status.idle": "2022-05-16T13:52:04.321105Z", "shell.execute_reply": "2022-05-16T13:52:04.320488Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:8000/fib/28 , 20 simultaneous requests, total time: 2.160097122192383\n" ] } ], "source": [ "# Before parallelism\n", "await measure('http://localhost:8000/fib/28', n=20)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:52:04.324603Z", "iopub.status.busy": "2022-05-16T13:52:04.324068Z", "iopub.status.idle": "2022-05-16T13:52:04.742722Z", "shell.execute_reply": "2022-05-16T13:52:04.742119Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:9000/fib/28 , 20 simultaneous requests, total time: 0.413372278213501\n" ] } ], "source": [ "# After parallelism\n", "await measure('http://localhost:9000/fib/28', n=20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Asynchronous computing\n", "\n", "Previously while one request was busy evaluating `fib(...)` Tornado was blocked. It was unable to handle any other request. This is particularly problematic when our server provides both expensive computations and cheap ones. The cheap requests get hung up needlessly.\n", "\n", "Because Dask is able to integrate with asynchronous systems like Tornado or Asyncio, our web server can freely jump between many requests, even while computation is going on in the background. In the example below we see that even though the slow computation started first, the fast computation returned in just a few milliseconds." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:52:04.745667Z", "iopub.status.busy": "2022-05-16T13:52:04.745263Z", "iopub.status.idle": "2022-05-16T13:52:07.884943Z", "shell.execute_reply": "2022-05-16T13:52:07.884073Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:8000/fib/35 , 1 simultaneous requests, total time: 3.1344475746154785\n", "http://localhost:8000/fast , 1 simultaneous requests, total time: 3.134436845779419\n" ] } ], "source": [ "# Before async\n", "a = asyncio.ensure_future(measure('http://localhost:8000/fib/35', n=1))\n", "b = asyncio.ensure_future(measure('http://localhost:8000/fast', n=1))\n", "await b\n", "await a" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:52:07.887776Z", "iopub.status.busy": "2022-05-16T13:52:07.887454Z", "iopub.status.idle": "2022-05-16T13:52:11.098161Z", "shell.execute_reply": "2022-05-16T13:52:11.097720Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:9000/fast , 1 simultaneous requests, total time: 0.009327888488769531\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "http://localhost:9000/fib/35 , 1 simultaneous requests, total time: 3.206266403198242\n" ] } ], "source": [ "# After async\n", "a = asyncio.ensure_future(measure('http://localhost:9000/fib/35', n=1))\n", "b = asyncio.ensure_future(measure('http://localhost:9000/fast', n=1))\n", "await b\n", "await a" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Other options\n", "\n", "In these situations people today tend to use [concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html) or [Celery](https://docs.celeryproject.org/en/latest/index.html).\n", "\n", "- concurrent.futures allows easy parallelism on a single machine and integrates well into async frameworks. The API is exactly what we showed above (Dask implements the concurrent.futures API). However concurrent.futures doesn't easily scale out to a cluster.\n", "- Celery scales out more easily to multiple machines, but has higher latencies, doesn't scale down as nicely, and needs a bit of effort to integrate into async frameworks (or at least this is my understanding, my experience here is shallow)\n", "\n", "In this context Dask provides some of the benefits of both. It is easy to set up and use in the common single-machine case, but can also [scale out to a cluster](http://distributed.readthedocs.io/en/latest/setup.html). It integrates nicely with async frameworks and adds only very small latencies." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-05-16T13:52:11.101149Z", "iopub.status.busy": "2022-05-16T13:52:11.100964Z", "iopub.status.idle": "2022-05-16T13:52:11.120185Z", "shell.execute_reply": "2022-05-16T13:52:11.119549Z" }, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Roundtrip latency: 14.59 ms\n" ] } ], "source": [ "async def f():\n", " start = time()\n", " result = await dask_client.submit(lambda x: x + 1, 10)\n", " end = time()\n", " print('Roundtrip latency: %.2f ms' % ((end - start) * 1000))\n", " \n", "await f()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 4 }