{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Handle Evolving Workflows\n", "=========================\n", "\n", "For some workflows we don't know the extent of the computation at the outset. We need to do some computation in order to figure out the rest of the computation that we need to do. The computation grows and evolves as we do more work.\n", "\n", "As an example, consider a situation where you need to read many files and then based on the contents of those files, fire off additional work. You would like to read the files in parallel, and then within each file expose more parallelism.\n", "\n", "This example goes through three ways to handle this situation using [Dask Futures](https://docs.dask.org/en/latest/futures.html)\n", "\n", "1. Using `as_completed`\n", "2. Using `async/await`\n", "3. Launching tasks from tasks\n", "\n", "But first, lets run our code sequentially." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "0: Sequential code\n", "------------------" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:37.152786Z", "iopub.status.busy": "2022-06-17T02:42:37.152513Z", "iopub.status.idle": "2022-06-17T02:42:37.169227Z", "shell.execute_reply": "2022-06-17T02:42:37.168582Z" } }, "outputs": [ { "data": { "text/plain": [ "['file.0.txt', 'file.1.txt', 'file.2.txt']" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filenames = [\"file.{}.txt\".format(i) for i in range(10)]\n", "\n", "filenames[:3]" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:37.172738Z", "iopub.status.busy": "2022-06-17T02:42:37.172389Z", "iopub.status.idle": "2022-06-17T02:42:37.178003Z", "shell.execute_reply": "2022-06-17T02:42:37.177266Z" } }, "outputs": [], "source": [ "import random, time\n", "\n", "\n", "def parse_file(fn: str) -> list:\n", " \"\"\" Returns a list work items of unknown length \"\"\"\n", " time.sleep(random.random())\n", " return [random.random() for _ in range(random.randint(1, 10))]\n", "\n", "def process_item(x: float):\n", " \"\"\" Process each work item \"\"\"\n", " time.sleep(random.random() / 4)\n", " return x + 1" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:37.181114Z", "iopub.status.busy": "2022-06-17T02:42:37.180689Z", "iopub.status.idle": "2022-06-17T02:42:49.210186Z", "shell.execute_reply": "2022-06-17T02:42:49.209167Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1.31 ms, sys: 6.64 ms, total: 7.95 ms\n", "Wall time: 12 s\n" ] } ], "source": [ "%%time\n", "\n", "# This takes around 10-20s\n", "\n", "results = []\n", "\n", "for fn in filenames:\n", " L = parse_file(fn)\n", " for x in L:\n", " out = process_item(x)\n", " results.append(out)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Start Dask Client\n", "-----------------\n", "\n", "We'll need a Dask client in order to manage dynamic workloads" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:49.213806Z", "iopub.status.busy": "2022-06-17T02:42:49.213379Z", "iopub.status.idle": "2022-06-17T02:42:50.389492Z", "shell.execute_reply": "2022-06-17T02:42:50.388720Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Client

\n", "

Client-2d67007f-ede7-11ec-a04b-000d3a5c8937

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
Connection method: Cluster objectCluster type: distributed.LocalCluster
\n", " Dashboard: http://10.1.0.7:8787/status\n", "
\n", "\n", " \n", "
\n", "

Cluster Info

\n", "
\n", "
\n", "
\n", "
\n", "

LocalCluster

\n", "

ce99b9df

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "\n", " \n", "
\n", " Dashboard: http://10.1.0.7:8787/status\n", " \n", " Workers: 1\n", "
\n", " Total threads: 6\n", " \n", " Total memory: 6.78 GiB\n", "
Status: runningUsing processes: False
\n", "\n", "
\n", " \n", "

Scheduler Info

\n", " \n", "\n", "
\n", "
\n", "
\n", "
\n", "

Scheduler

\n", "

Scheduler-ab93aa11-fa33-4438-99c7-f88383053d4e

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Comm: inproc://10.1.0.7/8267/1\n", " \n", " Workers: 1\n", "
\n", " Dashboard: http://10.1.0.7:8787/status\n", " \n", " Total threads: 6\n", "
\n", " Started: Just now\n", " \n", " Total memory: 6.78 GiB\n", "
\n", "
\n", "
\n", "\n", "
\n", " \n", "

Workers

\n", " \n", "\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: 0

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", "\n", "
\n", " Comm: inproc://10.1.0.7/8267/4\n", " \n", " Total threads: 6\n", "
\n", " Dashboard: http://10.1.0.7:40175/status\n", " \n", " Memory: 6.78 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /home/runner/work/dask-examples/dask-examples/applications/dask-worker-space/worker-18962zcv\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from dask.distributed import Client\n", "\n", "client = Client(processes=False, n_workers=1, threads_per_worker=6)\n", "client" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1: Use as_completed\n", "-------------------\n", "\n", "The [as_completed](https://docs.dask.org/en/latest/futures.html#distributed.as_completed) iterator lets us handle futures as they complete. We can then submit more data on the fly.\n", "\n", "- We submit a task for each of our filenames\n", "- We also compute the length of each of the returned lists\n", "- As those lengths return, we submit off a new task to get each item of that list. We do this at higher priority, so that we process existing data before we collect new data.\n", "- We wait on all of the returned results" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:50.392897Z", "iopub.status.busy": "2022-06-17T02:42:50.392498Z", "iopub.status.idle": "2022-06-17T02:42:53.117428Z", "shell.execute_reply": "2022-06-17T02:42:53.116509Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1.01 s, sys: 98.2 ms, total: 1.11 s\n", "Wall time: 2.71 s\n" ] }, { "data": { "text/plain": [ "[1.2518391185344768,\n", " 1.967300699173027,\n", " 1.1921768891007014,\n", " 1.1508851343978415,\n", " 1.0869530251966029,\n", " 1.3395127512611615,\n", " 1.8353113216002974,\n", " 1.7138099744257085,\n", " 1.6679971818055845,\n", " 1.8647842720985184,\n", " 1.3210153422733102,\n", " 1.5413927605930233,\n", " 1.164042907553493,\n", " 1.0314580101042854,\n", " 1.0711471685374874,\n", " 1.0442834426380299,\n", " 1.931199989512107,\n", " 1.3644475100733584,\n", " 1.9334433605591859,\n", " 1.9817491037589097,\n", " 1.406801443445758,\n", " 1.8909888768695662,\n", " 1.2837339021441623,\n", " 1.7395819078283443,\n", " 1.999778584542157,\n", " 1.6423609078708132,\n", " 1.396604708570918,\n", " 1.3887852336442459,\n", " 1.5926609545860728,\n", " 1.7886932827376854,\n", " 1.2265532338512402,\n", " 1.9138232304572873,\n", " 1.1197234843696666,\n", " 1.134131012517082,\n", " 1.5875756875849345,\n", " 1.6946853986324157,\n", " 1.7590876487763478,\n", " 1.2440401658236415,\n", " 1.4094107347747094,\n", " 1.6743308715242575,\n", " 1.0874906530079336,\n", " 1.3001027366763596,\n", " 1.7952312675387587,\n", " 1.3753142079981628,\n", " 1.9816492104338934,\n", " 1.1165130167276827,\n", " 1.5947212531479302,\n", " 1.5123508773551202,\n", " 1.4593046174051918,\n", " 1.9974620206967924,\n", " 1.0368492208645481,\n", " 1.056034920410224,\n", " 1.855376914140228,\n", " 1.8326127290601297,\n", " 1.5415864549382774,\n", " 1.7204088955479384,\n", " 1.1376171131203756,\n", " 1.7421486445457255,\n", " 1.2545887941884846,\n", " 1.4346396142463624,\n", " 1.7456176675184931,\n", " 1.6835317383609518,\n", " 1.414867990948455,\n", " 1.889774169465292,\n", " 1.713708648451582,\n", " 1.5212351343599329,\n", " 1.3773480118234862,\n", " 1.185082778813019,\n", " 1.6770678582874186,\n", " 1.6802326075034215,\n", " 1.3659486422397538]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "\n", "from dask.distributed import as_completed\n", "import operator\n", "\n", "lists = client.map(parse_file, filenames, pure=False)\n", "lengths = client.map(len, lists)\n", "\n", "mapping = dict(zip(lengths, lists))\n", "\n", "futures = []\n", "\n", "for future in as_completed(lengths):\n", " n = future.result()\n", " L = mapping[future]\n", " for i in range(n):\n", " new = client.submit(operator.getitem, L, i, priority=1)\n", " new = client.submit(process_item, new, priority=1)\n", " futures.append(new)\n", " \n", "client.gather(futures)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "2: Use async/await to handle single file processing locally\n", "-----------------------------------------------------------\n", "\n", "We can also handle the concurrency here within our local process. This requires you to understand async/await syntax, but is generally powerful and arguably simpler than the `as_completed` approach above." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:53.120456Z", "iopub.status.busy": "2022-06-17T02:42:53.120117Z", "iopub.status.idle": "2022-06-17T02:42:53.127554Z", "shell.execute_reply": "2022-06-17T02:42:53.126760Z" } }, "outputs": [], "source": [ "import asyncio\n", "\n", "async def f(fn):\n", " \"\"\" Handle the lifecycle of a single file \"\"\"\n", " future = client.submit(parse_file, fn, pure=False)\n", " length_future = client.submit(len, future)\n", " length = await length_future\n", " \n", " futures = [client.submit(operator.getitem, future, i, priority=10) \n", " for i in range(length)]\n", " futures = client.map(process_item, futures, priority=10)\n", " return futures\n", "\n", "async def run_all(filenames):\n", " list_of_list_of_futures = await asyncio.gather(*[f(fn) for fn in filenames])\n", " futures = sum(list_of_list_of_futures, [])\n", " return await client.gather(futures)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now need to run this function in the same event loop as our client is running. If we had started our client asynchronously, then we could have done this:\n", "\n", "```python\n", "client = await Client(asynchronous=True)\n", "\n", "await run_all(filenames)\n", "```\n", "\n", "However, because we started our client without the `asynchronous=True` flag the event loop is actually running in a separate thread, so we'll have to ask the client to run this for us." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:53.130242Z", "iopub.status.busy": "2022-06-17T02:42:53.130044Z", "iopub.status.idle": "2022-06-17T02:42:54.810510Z", "shell.execute_reply": "2022-06-17T02:42:54.809930Z" } }, "outputs": [ { "data": { "text/plain": [ "[1.1379214803897966,\n", " 1.8004468997619592,\n", " 1.1477710117536066,\n", " 1.1722491087770073,\n", " 1.2182080829905781,\n", " 1.1305521329516894,\n", " 1.5315118828105008,\n", " 1.4857109550716339,\n", " 1.1564552302047688,\n", " 1.7632671735712062,\n", " 1.7043367478356086,\n", " 1.6531882978486887,\n", " 1.6449568037500053,\n", " 1.1441160009442368,\n", " 1.8717558171360351,\n", " 1.3583440002967633,\n", " 1.693922800784076,\n", " 1.8780095173736684,\n", " 1.9653715196710575,\n", " 1.5787513479684643,\n", " 1.0882561072536192,\n", " 1.4834508964954152,\n", " 1.9824283555788558,\n", " 1.7626012062972949,\n", " 1.9614876283293154,\n", " 1.5538150550125693,\n", " 1.0835788240162896,\n", " 1.1465480726253297,\n", " 1.1279461112962195,\n", " 1.1496712116177383,\n", " 1.1477319237177976,\n", " 1.1999783254506513,\n", " 1.217967638377778,\n", " 1.0951703079673132,\n", " 1.0611877381374426,\n", " 1.1880676743279521,\n", " 1.0103101046672116,\n", " 1.3912816099354766]" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "client.sync(run_all, filenames)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "3: Submit tasks from tasks\n", "--------------------------\n", "\n", "We can also submit tasks that themselves submit more tasks. See [documentation here](https://docs.dask.org/en/latest/futures.html#submit-tasks-from-tasks)." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-06-17T02:42:54.813845Z", "iopub.status.busy": "2022-06-17T02:42:54.813460Z", "iopub.status.idle": "2022-06-17T02:42:57.359310Z", "shell.execute_reply": "2022-06-17T02:42:57.356200Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 310 ms, sys: 63.4 ms, total: 373 ms\n", "Wall time: 2.54 s\n" ] } ], "source": [ "%%time\n", "\n", "from dask.distributed import get_client, secede, rejoin\n", "\n", "def f(fn):\n", " L = parse_file(fn)\n", " client = get_client()\n", " \n", " futures = client.map(process_item, L, priority=10)\n", " secede()\n", " results = client.gather(futures)\n", " rejoin()\n", " return results\n", "\n", "futures = client.map(f, filenames, pure=False)\n", "results = client.gather(futures)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 4 }