{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dask Bags\n",
    "\n",
    "\n",
    "Dask Bag implements operations like `map`, `filter`, `groupby` and aggregations on collections of Python objects. It does this in parallel and in small memory using Python iterators. It is similar to a parallel version of itertools or a Pythonic version of the PySpark RDD.\n",
    "\n",
    "Dask Bags are often used to do simple preprocessing on log files, JSON records, or other user defined Python objects.\n",
    "\n",
    "Full API documentation is available here: http://docs.dask.org/en/latest/bag-api.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Dask Client for Dashboard\n",
    "\n",
    "Starting the Dask Client is optional.  It will provide a dashboard which \n",
    "is useful to gain insight on the computation.  \n",
    "\n",
    "The link to the dashboard will become visible when you create the client below.  We recommend having it open on one side of your screen while using your notebook on the other side.  This can take some effort to arrange your windows, but seeing them both at the same is very useful when learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:37.905858Z",
     "iopub.status.busy": "2021-10-14T16:43:37.904600Z",
     "iopub.status.idle": "2021-10-14T16:43:41.245946Z",
     "shell.execute_reply": "2021-10-14T16:43:41.246463Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Client</h3>\n",
       "<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n",
       "  <li><b>Scheduler: </b>tcp://127.0.0.1:38547</li>\n",
       "  <li><b>Dashboard: </b><a href='http://127.0.0.1:8787/status' target='_blank'>http://127.0.0.1:8787/status</a></li>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Cluster</h3>\n",
       "<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n",
       "  <li><b>Workers: </b>4</li>\n",
       "  <li><b>Cores: </b>4</li>\n",
       "  <li><b>Memory: </b>7.29 GB</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: 'tcp://127.0.0.1:38547' processes=4 threads=4, memory=7.29 GB>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dask.distributed import Client, progress\n",
    "client = Client(n_workers=4, threads_per_worker=1)\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Random Data\n",
    "\n",
    "We create a random set of record data and store it to disk as many JSON files.  This will serve as our data for this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:41.251840Z",
     "iopub.status.busy": "2021-10-14T16:43:41.251288Z",
     "iopub.status.idle": "2021-10-14T16:43:43.432598Z",
     "shell.execute_reply": "2021-10-14T16:43:43.432067Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/home/runner/work/dask-examples/dask-examples/data/0.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/1.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/2.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/3.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/4.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/5.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/6.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/7.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/8.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/9.json']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask\n",
    "import json\n",
    "import os\n",
    "\n",
    "os.makedirs('data', exist_ok=True)              # Create data/ directory\n",
    "\n",
    "b = dask.datasets.make_people()                 # Make records of people\n",
    "b.map(json.dumps).to_textfiles('data/*.json')   # Encode as JSON, write to disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read JSON data\n",
    "\n",
    "Now that we have some JSON data in a file lets take a look at it with Dask Bag and Python JSON module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.445397Z",
     "iopub.status.busy": "2021-10-14T16:43:43.444735Z",
     "iopub.status.idle": "2021-10-14T16:43:43.672691Z",
     "shell.execute_reply": "2021-10-14T16:43:43.671939Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"age\": 64, \"name\": [\"Maddie\", \"Collier\"], \"occupation\": \"Counsellor\", \"telephone\": \"1-649-913-3212\", \"address\": {\"address\": \"317 Mariner Garden\", \"city\": \"Carlsbad\"}, \"credit-card\": {\"number\": \"5597 2172 9760 3621\", \"expiration-date\": \"04/21\"}}\r\n",
      "{\"age\": 42, \"name\": [\"Alfred\", \"Gibson\"], \"occupation\": \"Lawyer\", \"telephone\": \"(010) 313-6325\", \"address\": {\"address\": \"995 Edgewood Stravenue\", \"city\": \"Lawndale\"}, \"credit-card\": {\"number\": \"3736 384667 05346\", \"expiration-date\": \"12/21\"}}\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 2 data/0.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.675788Z",
     "iopub.status.busy": "2021-10-14T16:43:43.675207Z",
     "iopub.status.idle": "2021-10-14T16:43:43.690028Z",
     "shell.execute_reply": "2021-10-14T16:43:43.690792Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dask.bag<loads, npartitions=10>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import dask.bag as db\n",
    "import json\n",
    "\n",
    "b = db.read_text('data/*.json').map(json.loads)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.693483Z",
     "iopub.status.busy": "2021-10-14T16:43:43.692939Z",
     "iopub.status.idle": "2021-10-14T16:43:43.721313Z",
     "shell.execute_reply": "2021-10-14T16:43:43.721908Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'age': 64,\n",
       "  'name': ['Maddie', 'Collier'],\n",
       "  'occupation': 'Counsellor',\n",
       "  'telephone': '1-649-913-3212',\n",
       "  'address': {'address': '317 Mariner Garden', 'city': 'Carlsbad'},\n",
       "  'credit-card': {'number': '5597 2172 9760 3621',\n",
       "   'expiration-date': '04/21'}},\n",
       " {'age': 42,\n",
       "  'name': ['Alfred', 'Gibson'],\n",
       "  'occupation': 'Lawyer',\n",
       "  'telephone': '(010) 313-6325',\n",
       "  'address': {'address': '995 Edgewood Stravenue', 'city': 'Lawndale'},\n",
       "  'credit-card': {'number': '3736 384667 05346', 'expiration-date': '12/21'}})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.take(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Map, Filter, Aggregate\n",
    "\n",
    "We can process this data by filtering out only certain records of interest, mapping functions over it to process our data, and aggregating those results to a total value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.724635Z",
     "iopub.status.busy": "2021-10-14T16:43:43.724028Z",
     "iopub.status.idle": "2021-10-14T16:43:43.751202Z",
     "shell.execute_reply": "2021-10-14T16:43:43.750699Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'age': 64,\n",
       "  'name': ['Maddie', 'Collier'],\n",
       "  'occupation': 'Counsellor',\n",
       "  'telephone': '1-649-913-3212',\n",
       "  'address': {'address': '317 Mariner Garden', 'city': 'Carlsbad'},\n",
       "  'credit-card': {'number': '5597 2172 9760 3621',\n",
       "   'expiration-date': '04/21'}},\n",
       " {'age': 42,\n",
       "  'name': ['Alfred', 'Gibson'],\n",
       "  'occupation': 'Lawyer',\n",
       "  'telephone': '(010) 313-6325',\n",
       "  'address': {'address': '995 Edgewood Stravenue', 'city': 'Lawndale'},\n",
       "  'credit-card': {'number': '3736 384667 05346', 'expiration-date': '12/21'}})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.filter(lambda record: record['age'] > 30).take(2)  # Select only people over 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.757428Z",
     "iopub.status.busy": "2021-10-14T16:43:43.755873Z",
     "iopub.status.idle": "2021-10-14T16:43:43.785836Z",
     "shell.execute_reply": "2021-10-14T16:43:43.784862Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Counsellor', 'Lawyer')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.map(lambda record: record['occupation']).take(2)  # Select the occupation field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.791648Z",
     "iopub.status.busy": "2021-10-14T16:43:43.791131Z",
     "iopub.status.idle": "2021-10-14T16:43:43.913496Z",
     "shell.execute_reply": "2021-10-14T16:43:43.912847Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.count().compute()  # Count total number of records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chain computations\n",
    "\n",
    "It is common to do many of these steps in one pipeline, only calling `compute` or `take` at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.919958Z",
     "iopub.status.busy": "2021-10-14T16:43:43.919390Z",
     "iopub.status.idle": "2021-10-14T16:43:43.922183Z",
     "shell.execute_reply": "2021-10-14T16:43:43.922783Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dask.bag<topk-aggregate, npartitions=1>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = (b.filter(lambda record: record['age'] > 30)\n",
    "           .map(lambda record: record['occupation'])\n",
    "           .frequencies(sort=True)\n",
    "           .topk(10, key=1))\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with all lazy Dask collections, we need to call `compute` to actually evaluate our result.  The `take` method used in earlier examples is also like `compute` and will also trigger computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:43.924914Z",
     "iopub.status.busy": "2021-10-14T16:43:43.924439Z",
     "iopub.status.idle": "2021-10-14T16:43:44.069667Z",
     "shell.execute_reply": "2021-10-14T16:43:44.069185Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Steel Worker', 15),\n",
       " ('Station Manager', 15),\n",
       " ('Vicar', 15),\n",
       " ('Calibration Manager', 15),\n",
       " ('Youth Worker', 15),\n",
       " ('Transport Engineer', 14),\n",
       " ('Professional Wrestler', 13),\n",
       " ('Hotel Consultant', 13),\n",
       " ('Bus Valeter', 13),\n",
       " ('Retired', 13)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform and Store\n",
    "\n",
    "Sometimes we want to compute aggregations as above, but sometimes we want to store results to disk for future analyses.  For that we can use methods like `to_textfiles` and `json.dumps`, or we can convert to Dask Dataframes and use their storage systems, which we'll see more of in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:44.072092Z",
     "iopub.status.busy": "2021-10-14T16:43:44.071534Z",
     "iopub.status.idle": "2021-10-14T16:43:44.238912Z",
     "shell.execute_reply": "2021-10-14T16:43:44.238319Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/home/runner/work/dask-examples/dask-examples/data/processed.0.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.1.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.2.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.3.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.4.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.5.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.6.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.7.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.8.json',\n",
       " '/home/runner/work/dask-examples/dask-examples/data/processed.9.json']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(b.filter(lambda record: record['age'] > 30)  # Select records of interest\n",
    "  .map(json.dumps)                            # Convert Python objects to text\n",
    "  .to_textfiles('data/processed.*.json'))     # Write to local disk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert to Dask Dataframes\n",
    "\n",
    "Dask Bags are good for reading in initial data, doing a bit of pre-processing, and then handing off to some other more efficient form like Dask Dataframes.  Dask Dataframes use Pandas internally, and so can be much faster on numeric data and also have more complex algorithms.  \n",
    "\n",
    "However, Dask Dataframes also expect data that is organized as flat columns.  It does not support nested JSON data very well (Bag is better for this).\n",
    "\n",
    "Here we make a function to flatten down our nested data structure, map that across our records, and then convert that to a Dask Dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:44.242904Z",
     "iopub.status.busy": "2021-10-14T16:43:44.242131Z",
     "iopub.status.idle": "2021-10-14T16:43:44.265766Z",
     "shell.execute_reply": "2021-10-14T16:43:44.265114Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'age': 64,\n",
       "  'name': ['Maddie', 'Collier'],\n",
       "  'occupation': 'Counsellor',\n",
       "  'telephone': '1-649-913-3212',\n",
       "  'address': {'address': '317 Mariner Garden', 'city': 'Carlsbad'},\n",
       "  'credit-card': {'number': '5597 2172 9760 3621',\n",
       "   'expiration-date': '04/21'}},)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:44.271774Z",
     "iopub.status.busy": "2021-10-14T16:43:44.271252Z",
     "iopub.status.idle": "2021-10-14T16:43:44.294736Z",
     "shell.execute_reply": "2021-10-14T16:43:44.295195Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'age': 64,\n",
       "  'occupation': 'Counsellor',\n",
       "  'telephone': '1-649-913-3212',\n",
       "  'credit-card-number': '5597 2172 9760 3621',\n",
       "  'credit-card-expiration': '04/21',\n",
       "  'name': 'Maddie Collier',\n",
       "  'street-address': '317 Mariner Garden',\n",
       "  'city': 'Carlsbad'},)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def flatten(record):\n",
    "    return {\n",
    "        'age': record['age'],\n",
    "        'occupation': record['occupation'],\n",
    "        'telephone': record['telephone'],\n",
    "        'credit-card-number': record['credit-card']['number'],\n",
    "        'credit-card-expiration': record['credit-card']['expiration-date'],\n",
    "        'name': ' '.join(record['name']),\n",
    "        'street-address': record['address']['address'],\n",
    "        'city': record['address']['city']   \n",
    "    }\n",
    "\n",
    "b.map(flatten).take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:44.297683Z",
     "iopub.status.busy": "2021-10-14T16:43:44.297038Z",
     "iopub.status.idle": "2021-10-14T16:43:45.214128Z",
     "shell.execute_reply": "2021-10-14T16:43:45.214542Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>telephone</th>\n",
       "      <th>credit-card-number</th>\n",
       "      <th>credit-card-expiration</th>\n",
       "      <th>name</th>\n",
       "      <th>street-address</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>64</td>\n",
       "      <td>Counsellor</td>\n",
       "      <td>1-649-913-3212</td>\n",
       "      <td>5597 2172 9760 3621</td>\n",
       "      <td>04/21</td>\n",
       "      <td>Maddie Collier</td>\n",
       "      <td>317 Mariner Garden</td>\n",
       "      <td>Carlsbad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>42</td>\n",
       "      <td>Lawyer</td>\n",
       "      <td>(010) 313-6325</td>\n",
       "      <td>3736 384667 05346</td>\n",
       "      <td>12/21</td>\n",
       "      <td>Alfred Gibson</td>\n",
       "      <td>995 Edgewood Stravenue</td>\n",
       "      <td>Lawndale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>Butchery Manager</td>\n",
       "      <td>622-149-4599</td>\n",
       "      <td>5158 0155 1512 5489</td>\n",
       "      <td>05/20</td>\n",
       "      <td>Antoine Bernard</td>\n",
       "      <td>31 Gough Point</td>\n",
       "      <td>Colleyville</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>64</td>\n",
       "      <td>Palaeobotanist</td>\n",
       "      <td>864.353.2333</td>\n",
       "      <td>5137 4444 9297 9682</td>\n",
       "      <td>09/20</td>\n",
       "      <td>Mario Hinton</td>\n",
       "      <td>410 Market Nene</td>\n",
       "      <td>Duncanville</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>Insurance Staff</td>\n",
       "      <td>1-751-176-1718</td>\n",
       "      <td>4899 9466 5476 4588</td>\n",
       "      <td>02/21</td>\n",
       "      <td>Walker Higgins</td>\n",
       "      <td>808 Fisher Wynd</td>\n",
       "      <td>Gardner</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age        occupation       telephone   credit-card-number  \\\n",
       "0   64        Counsellor  1-649-913-3212  5597 2172 9760 3621   \n",
       "1   42            Lawyer  (010) 313-6325    3736 384667 05346   \n",
       "2   60  Butchery Manager    622-149-4599  5158 0155 1512 5489   \n",
       "3   64    Palaeobotanist    864.353.2333  5137 4444 9297 9682   \n",
       "4   41   Insurance Staff  1-751-176-1718  4899 9466 5476 4588   \n",
       "\n",
       "  credit-card-expiration             name          street-address         city  \n",
       "0                  04/21   Maddie Collier      317 Mariner Garden     Carlsbad  \n",
       "1                  12/21    Alfred Gibson  995 Edgewood Stravenue     Lawndale  \n",
       "2                  05/20  Antoine Bernard          31 Gough Point  Colleyville  \n",
       "3                  09/20     Mario Hinton         410 Market Nene  Duncanville  \n",
       "4                  02/21   Walker Higgins         808 Fisher Wynd      Gardner  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = b.map(flatten).to_dataframe()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now perform the same computation as before, but now using Pandas and Dask dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-10-14T16:43:45.224210Z",
     "iopub.status.busy": "2021-10-14T16:43:45.223669Z",
     "iopub.status.idle": "2021-10-14T16:43:46.062051Z",
     "shell.execute_reply": "2021-10-14T16:43:46.060821Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Vicar                    15\n",
       "Steel Worker             15\n",
       "Youth Worker             15\n",
       "Calibration Manager      15\n",
       "Station Manager          15\n",
       "Transport Engineer       14\n",
       "Bus Valeter              13\n",
       "Retired                  13\n",
       "Editor                   13\n",
       "Professional Wrestler    13\n",
       "Name: occupation, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.age > 30].occupation.value_counts().nlargest(10).compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn More\n",
    "\n",
    "You may be interested in the following links:\n",
    "\n",
    "-  [Dask Bag Documentation](https://docs.dask.org/en/latest/bag.html)\n",
    "-  [API Documentation](http://docs.dask.org/en/latest/bag-api.html)\n",
    "-  [dask tutorial](https://github.com/dask/dask-tutorial), notebook 02, for a more in-depth introduction."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}