main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SQuAD Q&A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains training scripts for models to be used for the question answering problem on the [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) v1.1 dataset, which consists on selecting a possible answer to the given question as a span of words in the given context paragraph. The newest version (v2.0) of the dataset also contains unanswerable questions, but the one on which we worked on (v1.1) does not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Colab requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-18T15:12:30.609774Z",
     "iopub.status.busy": "2021-01-18T15:12:30.609319Z",
     "iopub.status.idle": "2021-01-18T15:12:30.701692Z"
    }
   },
   "source": [
    "Before restarting runtime (remember to select GPU runtime)$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/Wadaboa/squad-question-answering.git\n",
    "!pip install -r squad-question-answering/init/base_requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After restarting runtime$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, sys\n",
    "\n",
    "sys.path.insert(0, \"squad-question-answering\")\n",
    "os.chdir(\"squad-question-answering\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to import source files, we have to add the `src` folder to the Python `PATH`$\\dots$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-30T15:48:35.336945Z",
     "iopub.status.busy": "2021-01-30T15:48:35.336507Z",
     "iopub.status.idle": "2021-01-30T15:48:35.401035Z",
     "shell.execute_reply": "2021-01-30T15:48:35.399733Z",
     "shell.execute_reply.started": "2021-01-30T15:48:35.336899Z"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, \"src\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can import packages as usual$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:16:51.839836Z",
     "iopub.status.busy": "2021-02-01T08:16:51.839387Z",
     "iopub.status.idle": "2021-02-01T08:16:51.914841Z",
     "shell.execute_reply": "2021-02-01T08:16:51.913576Z",
     "shell.execute_reply.started": "2021-02-01T08:16:51.839789Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from functools import partial\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import wandb\n",
    "import transformers\n",
    "from transformers.trainer_utils import set_seed\n",
    "\n",
    "import dataset\n",
    "import model\n",
    "import training\n",
    "import tokenizer\n",
    "import utils\n",
    "import layer_utils\n",
    "import config\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the current configuration variables$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[(item, getattr(config, item)) for item in dir(config) if not item.startswith(\"__\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to perform some initialization stuff for all the libraries to be used throughout the notebook$\\dots$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weights & biases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training and evaluation metrics, along with model checkpoints and results, are directly logged into a [W&B](https://wandb.ai/) project, which is openly accessible [here](https://wandb.ai/wadaboa/squad-qa). Logging abilities are only granted to members of the team, so that if you want to launch your training run, you would have to disable wandb, by setting the environment variable `WANDB_DISABLED` to an empty value in the following block (`%env WANDB_DISABLED=`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:26.955252Z",
     "iopub.status.busy": "2021-01-24T17:19:26.954942Z",
     "iopub.status.idle": "2021-01-24T17:19:27.024560Z",
     "shell.execute_reply": "2021-01-24T17:19:27.023299Z",
     "shell.execute_reply.started": "2021-01-24T17:19:26.955215Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_PROJECT=squad-qa\n",
      "env: WANDB_ENTITY=wadaboa\n",
      "env: WANDB_MODE=online\n",
      "env: WANDB_RESUME=never\n",
      "env: WANDB_WATCH=false\n",
      "env: WANDB_SILENT=true\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_PROJECT=squad-qa\n",
    "%env WANDB_ENTITY=wadaboa\n",
    "%env WANDB_MODE=online\n",
    "%env WANDB_RESUME=never\n",
    "%env WANDB_WATCH=false\n",
    "%env WANDB_SILENT=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be sure to be logged in: if the system prompts you to insert a key, head over to [W&B](https://wandb.ai/authorize), login and the key should appear on the web page$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:29.550536Z",
     "iopub.status.busy": "2021-01-24T17:19:29.550147Z",
     "iopub.status.idle": "2021-01-24T17:19:31.670186Z",
     "shell.execute_reply": "2021-01-24T17:19:31.668397Z",
     "shell.execute_reply.started": "2021-01-24T17:19:29.550499Z"
    }
   },
   "outputs": [],
   "source": [
    "!wandb login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-26T10:54:28.223155Z",
     "iopub.status.busy": "2021-01-26T10:54:28.222723Z",
     "iopub.status.idle": "2021-01-26T10:54:28.289198Z",
     "shell.execute_reply": "2021-01-26T10:54:28.287720Z",
     "shell.execute_reply.started": "2021-01-26T10:54:28.223110Z"
    }
   },
   "source": [
    "Be sure to have `wandb` enabled system-wise$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:31.751590Z",
     "iopub.status.busy": "2021-01-24T17:19:31.751275Z",
     "iopub.status.idle": "2021-01-24T17:19:33.504344Z",
     "shell.execute_reply": "2021-01-24T17:19:33.502567Z",
     "shell.execute_reply.started": "2021-01-24T17:19:31.751553Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W&B enabled.\n"
     ]
    }
   ],
   "source": [
    "!wandb enabled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch and numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the random seed to a fixed number for reproducible results$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-30T16:25:59.206883Z",
     "iopub.status.busy": "2021-01-30T16:25:59.206469Z",
     "iopub.status.idle": "2021-01-30T16:25:59.271505Z",
     "shell.execute_reply": "2021-01-30T16:25:59.270311Z",
     "shell.execute_reply.started": "2021-01-30T16:25:59.206837Z"
    }
   },
   "outputs": [],
   "source": [
    "set_seed(config.RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the fastest device (GPU if available, else CPU as a fallback) to be used for training neural models in `PyTorch`$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-30T15:48:43.324496Z",
     "iopub.status.busy": "2021-01-30T15:48:43.324175Z",
     "iopub.status.idle": "2021-01-30T15:48:43.384976Z",
     "shell.execute_reply": "2021-01-30T15:48:43.383910Z",
     "shell.execute_reply.started": "2021-01-30T15:48:43.324456Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DEVICE = utils.get_device()\n",
    "DEVICE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a GPU device is available, print related info like GPU type, current usage$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-30T15:48:44.001464Z",
     "iopub.status.busy": "2021-01-30T15:48:44.001029Z",
     "iopub.status.idle": "2021-01-30T15:48:44.065379Z",
     "shell.execute_reply": "2021-01-30T15:48:44.064270Z",
     "shell.execute_reply.started": "2021-01-30T15:48:44.001418Z"
    }
   },
   "outputs": [],
   "source": [
    "if DEVICE.type != \"cpu\":\n",
    "    !nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to perform some preliminary steps, like data loading and common variables definition$\\dots$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SquadDataset` class holds a \"raw\" copy of the training set and the test set (if given). By \"raw\", we simply mean that questions and contexts are not pre-processed in this stage, but they are simply loaded from the given `JSON` files into appropriate `Pandas` `DataFrame`s.\n",
    "\n",
    "For the training set we used the official `SQuAD` v1.1 one, while for the test set we opted for the `SQuAD` v1.1 dev set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:16:59.492964Z",
     "iopub.status.busy": "2021-02-01T08:16:59.492515Z",
     "iopub.status.idle": "2021-02-01T08:16:59.559191Z",
     "shell.execute_reply": "2021-02-01T08:16:59.557910Z",
     "shell.execute_reply.started": "2021-02-01T08:16:59.492916Z"
    }
   },
   "outputs": [],
   "source": [
    "DATA_FOLDER = os.path.join(os.getcwd(), \"data\")\n",
    "TRAIN_DATA_FOLDER = os.path.join(DATA_FOLDER, \"training\")\n",
    "TRAIN_SET_PATH = os.path.join(TRAIN_DATA_FOLDER, \"training_set.json\")\n",
    "TEST_DATA_FOLDER = os.path.join(DATA_FOLDER, \"testing\")\n",
    "TEST_SET_PATH = os.path.join(TEST_DATA_FOLDER, \"test_set.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that the `subset` variable is used to load a random subset of both the training and testing dataset. This is to be used only for debugging purposes, so that `subset` should be set to $1.0$ when performing real training runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:17:00.010826Z",
     "iopub.status.busy": "2021-02-01T08:17:00.010511Z",
     "iopub.status.idle": "2021-02-01T08:17:00.277990Z",
     "shell.execute_reply": "2021-02-01T08:17:00.276675Z",
     "shell.execute_reply.started": "2021-02-01T08:17:00.010786Z"
    }
   },
   "outputs": [],
   "source": [
    "squad_dataset = dataset.SquadDataset(\n",
    "    train_set_path=TRAIN_SET_PATH,\n",
    "    test_set_path=TEST_SET_PATH,\n",
    "    subset=config.DATA_SUBSET,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the \"raw\" training set$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:17:00.891044Z",
     "iopub.status.busy": "2021-02-01T08:17:00.890721Z",
     "iopub.status.idle": "2021-02-01T08:17:00.989607Z",
     "shell.execute_reply": "2021-02-01T08:17:00.988084Z",
     "shell.execute_reply.started": "2021-02-01T08:17:00.891004Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer</th>\n",
       "      <th>title</th>\n",
       "      <th>context</th>\n",
       "      <th>question_id</th>\n",
       "      <th>question</th>\n",
       "      <th>context_id</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>167</td>\n",
       "      <td>1735</td>\n",
       "      <td>Institute_of_technology</td>\n",
       "      <td>The world's first institution of technology or...</td>\n",
       "      <td>56de4d9ecffd8e1900b4b7e2</td>\n",
       "      <td>What year was the Banská Akadémia founded?</td>\n",
       "      <td>1860</td>\n",
       "      <td>171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>793</td>\n",
       "      <td>SOS-based speed</td>\n",
       "      <td>Film_speed</td>\n",
       "      <td>The standard specifies how speed ratings shoul...</td>\n",
       "      <td>572674a05951b619008f7319</td>\n",
       "      <td>What is another speed that can also be reporte...</td>\n",
       "      <td>9354</td>\n",
       "      <td>808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>421</td>\n",
       "      <td>Sumerian temples and palaces</td>\n",
       "      <td>Sumer</td>\n",
       "      <td>The most impressive and famous of Sumerian bui...</td>\n",
       "      <td>5730bb058ab72b1400f9c72c</td>\n",
       "      <td>Where were the use of advanced materials and t...</td>\n",
       "      <td>17505</td>\n",
       "      <td>449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>192</td>\n",
       "      <td>mayor</td>\n",
       "      <td>Ann_Arbor,_Michigan</td>\n",
       "      <td>Ann Arbor has a council-manager form of govern...</td>\n",
       "      <td>572781a5f1498d1400e8fa1f</td>\n",
       "      <td>Who is elected every even numbered year?</td>\n",
       "      <td>10585</td>\n",
       "      <td>197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>194</td>\n",
       "      <td>decide on the feasibility of building an ICBM ...</td>\n",
       "      <td>John_von_Neumann</td>\n",
       "      <td>Shortly before his death, when he was already ...</td>\n",
       "      <td>572843ce4b864d190016485c</td>\n",
       "      <td>What was the purpose of top secret ICBM commit...</td>\n",
       "      <td>11497</td>\n",
       "      <td>284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>347</td>\n",
       "      <td>National Bishop Conferences</td>\n",
       "      <td>Pope_Paul_VI</td>\n",
       "      <td>Some critiqued Paul VI's decision; the newly c...</td>\n",
       "      <td>5726ef98708984140094d66e</td>\n",
       "      <td>What conferences became a requirement after Va...</td>\n",
       "      <td>10862</td>\n",
       "      <td>374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>105</td>\n",
       "      <td>C</td>\n",
       "      <td>Spectre_(2015_film)</td>\n",
       "      <td>Bond and Swann return to London where they mee...</td>\n",
       "      <td>56cdd28562d2951400fa68bd</td>\n",
       "      <td>Who does M fight with?</td>\n",
       "      <td>470</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>6</td>\n",
       "      <td>1150</td>\n",
       "      <td>Antarctica</td>\n",
       "      <td>About 1150 species of fungi have been recorded...</td>\n",
       "      <td>570e1a2a0dc6ce1900204dbf</td>\n",
       "      <td>How many species of fungi have been found on A...</td>\n",
       "      <td>6902</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>668</td>\n",
       "      <td>Virginia coastline</td>\n",
       "      <td>North_Carolina</td>\n",
       "      <td>In the Battle of Cowan's Ford, Cornwallis met ...</td>\n",
       "      <td>57278aa6f1498d1400e8fb65</td>\n",
       "      <td>After losing the battle of Guilford Courthouse...</td>\n",
       "      <td>12034</td>\n",
       "      <td>686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>128</td>\n",
       "      <td>aluminum.</td>\n",
       "      <td>2008_Summer_Olympics_torch_relay</td>\n",
       "      <td>The Olympic Torch is based on traditional scro...</td>\n",
       "      <td>56d8d4e7bfea0914004b7728</td>\n",
       "      <td>What is the Olympic Torch made from?</td>\n",
       "      <td>1456</td>\n",
       "      <td>137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>20</td>\n",
       "      <td>Somali National Television</td>\n",
       "      <td>Communications_in_Somalia</td>\n",
       "      <td>The Mogadishu-based Somali National Television...</td>\n",
       "      <td>56e1c12fe3433e140042312a</td>\n",
       "      <td>What TV station is the main public service bro...</td>\n",
       "      <td>3418</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>188</td>\n",
       "      <td>Armenian folk music</td>\n",
       "      <td>Armenians</td>\n",
       "      <td>Instruments like the duduk, the dhol, the zurn...</td>\n",
       "      <td>57324644e17f3d14004227b3</td>\n",
       "      <td>What kind of music does Sayat Nova play?</td>\n",
       "      <td>18452</td>\n",
       "      <td>207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>447</td>\n",
       "      <td>1311–1320</td>\n",
       "      <td>Sino-Tibetan_relations_during_the_Ming_dynasty</td>\n",
       "      <td>Nevertheless, the ethno-geographic caste hiera...</td>\n",
       "      <td>56ccf53362d2951400fa64fd</td>\n",
       "      <td>When did Ayurbarwada Buyantu Khan reign?</td>\n",
       "      <td>319</td>\n",
       "      <td>456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>452</td>\n",
       "      <td>Prussians were angered by what they considered...</td>\n",
       "      <td>Seven_Years%27_War</td>\n",
       "      <td>The war had also brought to an end the \"Old Sy...</td>\n",
       "      <td>572fc3ed04bcaa1900d76cb5</td>\n",
       "      <td>What drove Prussia away from renewing its alli...</td>\n",
       "      <td>15338</td>\n",
       "      <td>525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>158</td>\n",
       "      <td>Fourteen</td>\n",
       "      <td>Dissolution_of_the_Soviet_Union</td>\n",
       "      <td>On January 13, 1991, Soviet troops, along with...</td>\n",
       "      <td>57283caa3acd2414000df780</td>\n",
       "      <td>How many civilians died in the attack?</td>\n",
       "      <td>12205</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>490</td>\n",
       "      <td>Schenectady</td>\n",
       "      <td>General_Electric</td>\n",
       "      <td>At about the same time, Charles Coffin, leadin...</td>\n",
       "      <td>570d2a80b3d812140066d4d3</td>\n",
       "      <td>Which city was the home of GE's first headquar...</td>\n",
       "      <td>7281</td>\n",
       "      <td>501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>798</td>\n",
       "      <td>Red Stockings</td>\n",
       "      <td>Boston</td>\n",
       "      <td>The Boston Red Sox, a founding member of the A...</td>\n",
       "      <td>56e161bfcd28a01900c67849</td>\n",
       "      <td>What was the name of Bostons first baseball team?</td>\n",
       "      <td>3325</td>\n",
       "      <td>811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>571</td>\n",
       "      <td>2000s</td>\n",
       "      <td>Houston</td>\n",
       "      <td>One wave of the population boom ended abruptly...</td>\n",
       "      <td>5709b2a2200fba1400368279</td>\n",
       "      <td>When did Houston begin to regain its dependenc...</td>\n",
       "      <td>5914</td>\n",
       "      <td>576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>228</td>\n",
       "      <td>to gain air superiority</td>\n",
       "      <td>The_Blitz</td>\n",
       "      <td>Although not specifically prepared to conduct ...</td>\n",
       "      <td>572f785e04bcaa1900d769c0</td>\n",
       "      <td>Why did the Luftwaffe bomb the RAF Fighter Com...</td>\n",
       "      <td>15807</td>\n",
       "      <td>251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>455</td>\n",
       "      <td>they act as tradeoffs such, whereby to be more...</td>\n",
       "      <td>Sexual_orientation</td>\n",
       "      <td>A third concern with the Kinsey scale is that ...</td>\n",
       "      <td>570fa4675ab6b81900390f5f</td>\n",
       "      <td>What happens if the concepts are measured on t...</td>\n",
       "      <td>7632</td>\n",
       "      <td>554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>967</td>\n",
       "      <td>300 acres</td>\n",
       "      <td>Ann_Arbor,_Michigan</td>\n",
       "      <td>Ann Arbor's \"Tree Town\" nickname stems from th...</td>\n",
       "      <td>572755d8708984140094dc5a</td>\n",
       "      <td>How big is the Matthaei botanical garden?</td>\n",
       "      <td>10568</td>\n",
       "      <td>976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>158</td>\n",
       "      <td>spatial redundancy reduction</td>\n",
       "      <td>Data_compression</td>\n",
       "      <td>Today, nearly all commonly used video compress...</td>\n",
       "      <td>57268b4d5951b619008f7640</td>\n",
       "      <td>What does a DCT do?</td>\n",
       "      <td>10310</td>\n",
       "      <td>186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>668</td>\n",
       "      <td>1978</td>\n",
       "      <td>Post-punk</td>\n",
       "      <td>Also emerging during this period was New York'...</td>\n",
       "      <td>572f781404bcaa1900d769b8</td>\n",
       "      <td>When was ZE Records founded?</td>\n",
       "      <td>15248</td>\n",
       "      <td>672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>483</td>\n",
       "      <td>MTV</td>\n",
       "      <td>Madonna_(entertainer)</td>\n",
       "      <td>Academics noted that with her videos, Madonna ...</td>\n",
       "      <td>5726fbec708984140094d7ad</td>\n",
       "      <td>Who named Madonna the Greatest Music Video sta...</td>\n",
       "      <td>10034</td>\n",
       "      <td>486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>389</td>\n",
       "      <td>Liu Bei</td>\n",
       "      <td>Nanjing</td>\n",
       "      <td>Surrounded by the Yangtze River and mountains,...</td>\n",
       "      <td>56e79e2e00c9c71400d773d5</td>\n",
       "      <td>Who convinced Sun Quan to make Nanjing his cap...</td>\n",
       "      <td>3759</td>\n",
       "      <td>396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>197</td>\n",
       "      <td>operating its own customs and quarantine agency</td>\n",
       "      <td>Guam</td>\n",
       "      <td>Guam is served by the Antonio B. Won Pat Inter...</td>\n",
       "      <td>572b63a8be1ee31400cb834d</td>\n",
       "      <td>What is Guam responsible for when goods both c...</td>\n",
       "      <td>15089</td>\n",
       "      <td>244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>15</td>\n",
       "      <td>Medicines and Healthcare Products Regulatory A...</td>\n",
       "      <td>Pharmaceutical_industry</td>\n",
       "      <td>In the UK, the Medicines and Healthcare Produc...</td>\n",
       "      <td>571af86f32177014007ea003</td>\n",
       "      <td>Who is responsible for approving drugs in the ...</td>\n",
       "      <td>8064</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>588</td>\n",
       "      <td>the \"sick man\"</td>\n",
       "      <td>Ottoman_Empire</td>\n",
       "      <td>The Serbian revolution (1804–1815) marked the ...</td>\n",
       "      <td>572a301c3f37b31900478791</td>\n",
       "      <td>What did Europeans refer to the Ottoman empire...</td>\n",
       "      <td>14726</td>\n",
       "      <td>602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>399</td>\n",
       "      <td>Mexico</td>\n",
       "      <td>Dwight_D._Eisenhower</td>\n",
       "      <td>In 1954, Eisenhower articulated the domino the...</td>\n",
       "      <td>57326b2be99e3014001e678a</td>\n",
       "      <td>What country did Eisenhower believe communists...</td>\n",
       "      <td>18556</td>\n",
       "      <td>405</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>400</td>\n",
       "      <td>He thought it reasonable that species with an ...</td>\n",
       "      <td>On_the_Origin_of_Species</td>\n",
       "      <td>Chapter VII (of the first edition) addresses t...</td>\n",
       "      <td>5727b8b3ff5b5019007d936a</td>\n",
       "      <td>How does Darwin theorize that instincts have e...</td>\n",
       "      <td>12124</td>\n",
       "      <td>515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>164</td>\n",
       "      <td>Hellas</td>\n",
       "      <td>Greece</td>\n",
       "      <td>The names for the nation of Greece and the Gre...</td>\n",
       "      <td>57261a26ec44d21400f3d8dd</td>\n",
       "      <td>What is one of the names the Greeks call their...</td>\n",
       "      <td>9096</td>\n",
       "      <td>170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>55</td>\n",
       "      <td>1939</td>\n",
       "      <td>Tajikistan</td>\n",
       "      <td>Tajiks began to be conscripted into the Soviet...</td>\n",
       "      <td>5733acebd058e614000b600b</td>\n",
       "      <td>When did Tajiks start being part of the Soviet...</td>\n",
       "      <td>846</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>210</td>\n",
       "      <td>1867</td>\n",
       "      <td>Russian_language</td>\n",
       "      <td>The language was first introduced in North Ame...</td>\n",
       "      <td>5730e7c0b54a4f140068ccd2</td>\n",
       "      <td>When did the US buy Alaska?</td>\n",
       "      <td>17646</td>\n",
       "      <td>214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>742</td>\n",
       "      <td>41 engineering colleges</td>\n",
       "      <td>Rajasthan</td>\n",
       "      <td>In Rajasthan, Jodhpur and Kota are two major e...</td>\n",
       "      <td>572ab10734ae481900deac33</td>\n",
       "      <td>How many engineering colleges are in Rajasthan?</td>\n",
       "      <td>15059</td>\n",
       "      <td>765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>349</td>\n",
       "      <td>hostile aircraft flying over Britain</td>\n",
       "      <td>Anti-aircraft_warfare</td>\n",
       "      <td>From the early 1930s eight countries developed...</td>\n",
       "      <td>570d3f2ffed7b91900d45d98</td>\n",
       "      <td>What did the Observer Corps observe and report...</td>\n",
       "      <td>7140</td>\n",
       "      <td>385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>11</td>\n",
       "      <td>seven</td>\n",
       "      <td>USB</td>\n",
       "      <td>A group of seven companies began the developme...</td>\n",
       "      <td>5727b0ccff5b5019007d92a9</td>\n",
       "      <td>How many companies developed USB's?</td>\n",
       "      <td>12767</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>61</td>\n",
       "      <td>House Master</td>\n",
       "      <td>Eton_College</td>\n",
       "      <td>The primary responsibility for a boy's studies...</td>\n",
       "      <td>5727be06ff5b5019007d93fd</td>\n",
       "      <td>With whom does the primary responsibility for ...</td>\n",
       "      <td>12592</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>102</td>\n",
       "      <td>wire filament</td>\n",
       "      <td>Incandescent_light_bulb</td>\n",
       "      <td>An incandescent light bulb, incandescent lamp ...</td>\n",
       "      <td>5725c70dec44d21400f3d540</td>\n",
       "      <td>Which part of the incandescent light bulb is h...</td>\n",
       "      <td>8805</td>\n",
       "      <td>115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>72</td>\n",
       "      <td>200,000</td>\n",
       "      <td>New_York_City</td>\n",
       "      <td>The Great Irish Famine brought a large influx ...</td>\n",
       "      <td>56cee5a1aab44d1400b88c24</td>\n",
       "      <td>In 1860, approximately how many people of Iris...</td>\n",
       "      <td>605</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>50</td>\n",
       "      <td>Oklahoma City</td>\n",
       "      <td>Kanye_West</td>\n",
       "      <td>The funeral and burial for Donda West was held...</td>\n",
       "      <td>56cf83a7234ae51400d9bde3</td>\n",
       "      <td>Where was Donda West's funeral?</td>\n",
       "      <td>1083</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>690</td>\n",
       "      <td>1,000 households</td>\n",
       "      <td>Sino-Tibetan_relations_during_the_Ming_dynasty</td>\n",
       "      <td>Chen Qingying, Professor of History and Direct...</td>\n",
       "      <td>56cd480b62d2951400fa6510</td>\n",
       "      <td>How many households were the offices of Qianhu...</td>\n",
       "      <td>321</td>\n",
       "      <td>706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>457</td>\n",
       "      <td>1941</td>\n",
       "      <td>The_Blitz</td>\n",
       "      <td>Communal shelters never housed more than one s...</td>\n",
       "      <td>572fabd004bcaa1900d76bac</td>\n",
       "      <td>What year did the government start giving out ...</td>\n",
       "      <td>15820</td>\n",
       "      <td>461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>568</td>\n",
       "      <td>he came to the defense of a student who was in...</td>\n",
       "      <td>Yale_University</td>\n",
       "      <td>Yale has a history of difficult and prolonged ...</td>\n",
       "      <td>5726dde0f1498d1400e8edf7</td>\n",
       "      <td>Why was Professor David Graeber retired during...</td>\n",
       "      <td>10500</td>\n",
       "      <td>644</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    answer_start                                             answer  \\\n",
       "0            167                                               1735   \n",
       "1            793                                    SOS-based speed   \n",
       "2            421                       Sumerian temples and palaces   \n",
       "3            192                                              mayor   \n",
       "4            194  decide on the feasibility of building an ICBM ...   \n",
       "5            347                        National Bishop Conferences   \n",
       "6            105                                                  C   \n",
       "7              6                                               1150   \n",
       "8            668                                 Virginia coastline   \n",
       "9            128                                          aluminum.   \n",
       "10            20                         Somali National Television   \n",
       "11           188                                Armenian folk music   \n",
       "12           447                                          1311–1320   \n",
       "13           452  Prussians were angered by what they considered...   \n",
       "14           158                                           Fourteen   \n",
       "15           490                                        Schenectady   \n",
       "16           798                                      Red Stockings   \n",
       "17           571                                              2000s   \n",
       "18           228                            to gain air superiority   \n",
       "19           455  they act as tradeoffs such, whereby to be more...   \n",
       "20           967                                          300 acres   \n",
       "21           158                       spatial redundancy reduction   \n",
       "22           668                                               1978   \n",
       "23           483                                                MTV   \n",
       "24           389                                            Liu Bei   \n",
       "25           197    operating its own customs and quarantine agency   \n",
       "26            15  Medicines and Healthcare Products Regulatory A...   \n",
       "27           588                                     the \"sick man\"   \n",
       "28           399                                             Mexico   \n",
       "29           400  He thought it reasonable that species with an ...   \n",
       "30           164                                             Hellas   \n",
       "31            55                                               1939   \n",
       "32           210                                               1867   \n",
       "33           742                            41 engineering colleges   \n",
       "34           349               hostile aircraft flying over Britain   \n",
       "35            11                                              seven   \n",
       "36            61                                       House Master   \n",
       "37           102                                      wire filament   \n",
       "38            72                                            200,000   \n",
       "39            50                                      Oklahoma City   \n",
       "40           690                                   1,000 households   \n",
       "41           457                                               1941   \n",
       "42           568  he came to the defense of a student who was in...   \n",
       "\n",
       "                                             title  \\\n",
       "0                          Institute_of_technology   \n",
       "1                                       Film_speed   \n",
       "2                                            Sumer   \n",
       "3                              Ann_Arbor,_Michigan   \n",
       "4                                 John_von_Neumann   \n",
       "5                                     Pope_Paul_VI   \n",
       "6                              Spectre_(2015_film)   \n",
       "7                                       Antarctica   \n",
       "8                                   North_Carolina   \n",
       "9                 2008_Summer_Olympics_torch_relay   \n",
       "10                       Communications_in_Somalia   \n",
       "11                                       Armenians   \n",
       "12  Sino-Tibetan_relations_during_the_Ming_dynasty   \n",
       "13                              Seven_Years%27_War   \n",
       "14                 Dissolution_of_the_Soviet_Union   \n",
       "15                                General_Electric   \n",
       "16                                          Boston   \n",
       "17                                         Houston   \n",
       "18                                       The_Blitz   \n",
       "19                              Sexual_orientation   \n",
       "20                             Ann_Arbor,_Michigan   \n",
       "21                                Data_compression   \n",
       "22                                       Post-punk   \n",
       "23                           Madonna_(entertainer)   \n",
       "24                                         Nanjing   \n",
       "25                                            Guam   \n",
       "26                         Pharmaceutical_industry   \n",
       "27                                  Ottoman_Empire   \n",
       "28                            Dwight_D._Eisenhower   \n",
       "29                        On_the_Origin_of_Species   \n",
       "30                                          Greece   \n",
       "31                                      Tajikistan   \n",
       "32                                Russian_language   \n",
       "33                                       Rajasthan   \n",
       "34                           Anti-aircraft_warfare   \n",
       "35                                             USB   \n",
       "36                                    Eton_College   \n",
       "37                         Incandescent_light_bulb   \n",
       "38                                   New_York_City   \n",
       "39                                      Kanye_West   \n",
       "40  Sino-Tibetan_relations_during_the_Ming_dynasty   \n",
       "41                                       The_Blitz   \n",
       "42                                 Yale_University   \n",
       "\n",
       "                                              context  \\\n",
       "0   The world's first institution of technology or...   \n",
       "1   The standard specifies how speed ratings shoul...   \n",
       "2   The most impressive and famous of Sumerian bui...   \n",
       "3   Ann Arbor has a council-manager form of govern...   \n",
       "4   Shortly before his death, when he was already ...   \n",
       "5   Some critiqued Paul VI's decision; the newly c...   \n",
       "6   Bond and Swann return to London where they mee...   \n",
       "7   About 1150 species of fungi have been recorded...   \n",
       "8   In the Battle of Cowan's Ford, Cornwallis met ...   \n",
       "9   The Olympic Torch is based on traditional scro...   \n",
       "10  The Mogadishu-based Somali National Television...   \n",
       "11  Instruments like the duduk, the dhol, the zurn...   \n",
       "12  Nevertheless, the ethno-geographic caste hiera...   \n",
       "13  The war had also brought to an end the \"Old Sy...   \n",
       "14  On January 13, 1991, Soviet troops, along with...   \n",
       "15  At about the same time, Charles Coffin, leadin...   \n",
       "16  The Boston Red Sox, a founding member of the A...   \n",
       "17  One wave of the population boom ended abruptly...   \n",
       "18  Although not specifically prepared to conduct ...   \n",
       "19  A third concern with the Kinsey scale is that ...   \n",
       "20  Ann Arbor's \"Tree Town\" nickname stems from th...   \n",
       "21  Today, nearly all commonly used video compress...   \n",
       "22  Also emerging during this period was New York'...   \n",
       "23  Academics noted that with her videos, Madonna ...   \n",
       "24  Surrounded by the Yangtze River and mountains,...   \n",
       "25  Guam is served by the Antonio B. Won Pat Inter...   \n",
       "26  In the UK, the Medicines and Healthcare Produc...   \n",
       "27  The Serbian revolution (1804–1815) marked the ...   \n",
       "28  In 1954, Eisenhower articulated the domino the...   \n",
       "29  Chapter VII (of the first edition) addresses t...   \n",
       "30  The names for the nation of Greece and the Gre...   \n",
       "31  Tajiks began to be conscripted into the Soviet...   \n",
       "32  The language was first introduced in North Ame...   \n",
       "33  In Rajasthan, Jodhpur and Kota are two major e...   \n",
       "34  From the early 1930s eight countries developed...   \n",
       "35  A group of seven companies began the developme...   \n",
       "36  The primary responsibility for a boy's studies...   \n",
       "37  An incandescent light bulb, incandescent lamp ...   \n",
       "38  The Great Irish Famine brought a large influx ...   \n",
       "39  The funeral and burial for Donda West was held...   \n",
       "40  Chen Qingying, Professor of History and Direct...   \n",
       "41  Communal shelters never housed more than one s...   \n",
       "42  Yale has a history of difficult and prolonged ...   \n",
       "\n",
       "                 question_id  \\\n",
       "0   56de4d9ecffd8e1900b4b7e2   \n",
       "1   572674a05951b619008f7319   \n",
       "2   5730bb058ab72b1400f9c72c   \n",
       "3   572781a5f1498d1400e8fa1f   \n",
       "4   572843ce4b864d190016485c   \n",
       "5   5726ef98708984140094d66e   \n",
       "6   56cdd28562d2951400fa68bd   \n",
       "7   570e1a2a0dc6ce1900204dbf   \n",
       "8   57278aa6f1498d1400e8fb65   \n",
       "9   56d8d4e7bfea0914004b7728   \n",
       "10  56e1c12fe3433e140042312a   \n",
       "11  57324644e17f3d14004227b3   \n",
       "12  56ccf53362d2951400fa64fd   \n",
       "13  572fc3ed04bcaa1900d76cb5   \n",
       "14  57283caa3acd2414000df780   \n",
       "15  570d2a80b3d812140066d4d3   \n",
       "16  56e161bfcd28a01900c67849   \n",
       "17  5709b2a2200fba1400368279   \n",
       "18  572f785e04bcaa1900d769c0   \n",
       "19  570fa4675ab6b81900390f5f   \n",
       "20  572755d8708984140094dc5a   \n",
       "21  57268b4d5951b619008f7640   \n",
       "22  572f781404bcaa1900d769b8   \n",
       "23  5726fbec708984140094d7ad   \n",
       "24  56e79e2e00c9c71400d773d5   \n",
       "25  572b63a8be1ee31400cb834d   \n",
       "26  571af86f32177014007ea003   \n",
       "27  572a301c3f37b31900478791   \n",
       "28  57326b2be99e3014001e678a   \n",
       "29  5727b8b3ff5b5019007d936a   \n",
       "30  57261a26ec44d21400f3d8dd   \n",
       "31  5733acebd058e614000b600b   \n",
       "32  5730e7c0b54a4f140068ccd2   \n",
       "33  572ab10734ae481900deac33   \n",
       "34  570d3f2ffed7b91900d45d98   \n",
       "35  5727b0ccff5b5019007d92a9   \n",
       "36  5727be06ff5b5019007d93fd   \n",
       "37  5725c70dec44d21400f3d540   \n",
       "38  56cee5a1aab44d1400b88c24   \n",
       "39  56cf83a7234ae51400d9bde3   \n",
       "40  56cd480b62d2951400fa6510   \n",
       "41  572fabd004bcaa1900d76bac   \n",
       "42  5726dde0f1498d1400e8edf7   \n",
       "\n",
       "                                             question  context_id  answer_end  \n",
       "0          What year was the Banská Akadémia founded?        1860         171  \n",
       "1   What is another speed that can also be reporte...        9354         808  \n",
       "2   Where were the use of advanced materials and t...       17505         449  \n",
       "3            Who is elected every even numbered year?       10585         197  \n",
       "4   What was the purpose of top secret ICBM commit...       11497         284  \n",
       "5   What conferences became a requirement after Va...       10862         374  \n",
       "6                              Who does M fight with?         470         106  \n",
       "7   How many species of fungi have been found on A...        6902          10  \n",
       "8   After losing the battle of Guilford Courthouse...       12034         686  \n",
       "9                What is the Olympic Torch made from?        1456         137  \n",
       "10  What TV station is the main public service bro...        3418          46  \n",
       "11           What kind of music does Sayat Nova play?       18452         207  \n",
       "12           When did Ayurbarwada Buyantu Khan reign?         319         456  \n",
       "13  What drove Prussia away from renewing its alli...       15338         525  \n",
       "14             How many civilians died in the attack?       12205         166  \n",
       "15  Which city was the home of GE's first headquar...        7281         501  \n",
       "16  What was the name of Bostons first baseball team?        3325         811  \n",
       "17  When did Houston begin to regain its dependenc...        5914         576  \n",
       "18  Why did the Luftwaffe bomb the RAF Fighter Com...       15807         251  \n",
       "19  What happens if the concepts are measured on t...        7632         554  \n",
       "20          How big is the Matthaei botanical garden?       10568         976  \n",
       "21                                What does a DCT do?       10310         186  \n",
       "22                       When was ZE Records founded?       15248         672  \n",
       "23  Who named Madonna the Greatest Music Video sta...       10034         486  \n",
       "24  Who convinced Sun Quan to make Nanjing his cap...        3759         396  \n",
       "25  What is Guam responsible for when goods both c...       15089         244  \n",
       "26  Who is responsible for approving drugs in the ...        8064          66  \n",
       "27  What did Europeans refer to the Ottoman empire...       14726         602  \n",
       "28  What country did Eisenhower believe communists...       18556         405  \n",
       "29  How does Darwin theorize that instincts have e...       12124         515  \n",
       "30  What is one of the names the Greeks call their...        9096         170  \n",
       "31  When did Tajiks start being part of the Soviet...         846          59  \n",
       "32                        When did the US buy Alaska?       17646         214  \n",
       "33    How many engineering colleges are in Rajasthan?       15059         765  \n",
       "34  What did the Observer Corps observe and report...        7140         385  \n",
       "35                How many companies developed USB's?       12767          16  \n",
       "36  With whom does the primary responsibility for ...       12592          73  \n",
       "37  Which part of the incandescent light bulb is h...        8805         115  \n",
       "38  In 1860, approximately how many people of Iris...         605          79  \n",
       "39                    Where was Donda West's funeral?        1083          63  \n",
       "40  How many households were the offices of Qianhu...         321         706  \n",
       "41  What year did the government start giving out ...       15820         461  \n",
       "42  Why was Professor David Graeber retired during...       10500         644  "
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "squad_dataset.raw_train_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the \"raw\" test set$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:17:02.115526Z",
     "iopub.status.busy": "2021-02-01T08:17:02.115077Z",
     "iopub.status.idle": "2021-02-01T08:17:02.197654Z",
     "shell.execute_reply": "2021-02-01T08:17:02.196425Z",
     "shell.execute_reply.started": "2021-02-01T08:17:02.115479Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer</th>\n",
       "      <th>title</th>\n",
       "      <th>context</th>\n",
       "      <th>question_id</th>\n",
       "      <th>question</th>\n",
       "      <th>context_id</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>250</td>\n",
       "      <td>instantaneously in action-reaction pairs</td>\n",
       "      <td>Force</td>\n",
       "      <td>Tension forces can be modeled using ideal stri...</td>\n",
       "      <td>57379ed81c456719005744d7</td>\n",
       "      <td>In what way do idea strings transmit tesion fo...</td>\n",
       "      <td>2058</td>\n",
       "      <td>290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>187</td>\n",
       "      <td>Egg of Columbus</td>\n",
       "      <td>Nikola_Tesla</td>\n",
       "      <td>Tesla also explained the principles of the rot...</td>\n",
       "      <td>56e0ed557aa994140058e7dd</td>\n",
       "      <td>What was Tesla's device called?</td>\n",
       "      <td>181</td>\n",
       "      <td>202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>539</td>\n",
       "      <td>NP-complete Boolean satisfiability problem</td>\n",
       "      <td>Computational_complexity_theory</td>\n",
       "      <td>What intractability means in practice is open ...</td>\n",
       "      <td>56e1febfe3433e140042323a</td>\n",
       "      <td>What is the example of another problem charact...</td>\n",
       "      <td>282</td>\n",
       "      <td>581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1332</td>\n",
       "      <td>Treaty provisions</td>\n",
       "      <td>European_Union_law</td>\n",
       "      <td>Although it is generally accepted that EU law ...</td>\n",
       "      <td>57269bb8708984140094cb98</td>\n",
       "      <td>What are EU Regulations essentially the same a...</td>\n",
       "      <td>759</td>\n",
       "      <td>1349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>181</td>\n",
       "      <td>Gerhard</td>\n",
       "      <td>Martin_Luther</td>\n",
       "      <td>The Lutheran theologian Franz Pieper observed ...</td>\n",
       "      <td>56f884cba6d7ea1400e17708</td>\n",
       "      <td>What theologian differed in views about the so...</td>\n",
       "      <td>412</td>\n",
       "      <td>188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1441</td>\n",
       "      <td>self-consistent unification</td>\n",
       "      <td>Force</td>\n",
       "      <td>The development of fundamental theories for fo...</td>\n",
       "      <td>5737821cc3c5551400e51f1c</td>\n",
       "      <td>What type of physics model did Einstein fail t...</td>\n",
       "      <td>2045</td>\n",
       "      <td>1468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>503</td>\n",
       "      <td>not a unit and cannot be written as a product ...</td>\n",
       "      <td>Prime_number</td>\n",
       "      <td>Prime numbers give rise to two more general co...</td>\n",
       "      <td>57299c2c6aef051400155024</td>\n",
       "      <td>Under what condition is an element irreducible?</td>\n",
       "      <td>1758</td>\n",
       "      <td>589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>221</td>\n",
       "      <td>the filling of molecular orbitals formed from ...</td>\n",
       "      <td>Oxygen</td>\n",
       "      <td>In this dioxygen, the two oxygen atoms are che...</td>\n",
       "      <td>571c83f3dd7acb1400e4c0dc</td>\n",
       "      <td>Of what does the covalent double bond result f...</td>\n",
       "      <td>630</td>\n",
       "      <td>317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>11</td>\n",
       "      <td>pupils are free to choose a private school</td>\n",
       "      <td>Private_school</td>\n",
       "      <td>In Sweden, pupils are free to choose a private...</td>\n",
       "      <td>572754dd708984140094dc3f</td>\n",
       "      <td>What school model is Sweden notable for?</td>\n",
       "      <td>1343</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   answer_start                                             answer  \\\n",
       "0           250           instantaneously in action-reaction pairs   \n",
       "1           187                                    Egg of Columbus   \n",
       "2           539         NP-complete Boolean satisfiability problem   \n",
       "3          1332                                  Treaty provisions   \n",
       "4           181                                            Gerhard   \n",
       "5          1441                        self-consistent unification   \n",
       "6           503  not a unit and cannot be written as a product ...   \n",
       "7           221  the filling of molecular orbitals formed from ...   \n",
       "8            11         pupils are free to choose a private school   \n",
       "\n",
       "                             title  \\\n",
       "0                            Force   \n",
       "1                     Nikola_Tesla   \n",
       "2  Computational_complexity_theory   \n",
       "3               European_Union_law   \n",
       "4                    Martin_Luther   \n",
       "5                            Force   \n",
       "6                     Prime_number   \n",
       "7                           Oxygen   \n",
       "8                   Private_school   \n",
       "\n",
       "                                             context  \\\n",
       "0  Tension forces can be modeled using ideal stri...   \n",
       "1  Tesla also explained the principles of the rot...   \n",
       "2  What intractability means in practice is open ...   \n",
       "3  Although it is generally accepted that EU law ...   \n",
       "4  The Lutheran theologian Franz Pieper observed ...   \n",
       "5  The development of fundamental theories for fo...   \n",
       "6  Prime numbers give rise to two more general co...   \n",
       "7  In this dioxygen, the two oxygen atoms are che...   \n",
       "8  In Sweden, pupils are free to choose a private...   \n",
       "\n",
       "                question_id  \\\n",
       "0  57379ed81c456719005744d7   \n",
       "1  56e0ed557aa994140058e7dd   \n",
       "2  56e1febfe3433e140042323a   \n",
       "3  57269bb8708984140094cb98   \n",
       "4  56f884cba6d7ea1400e17708   \n",
       "5  5737821cc3c5551400e51f1c   \n",
       "6  57299c2c6aef051400155024   \n",
       "7  571c83f3dd7acb1400e4c0dc   \n",
       "8  572754dd708984140094dc3f   \n",
       "\n",
       "                                            question  context_id  answer_end  \n",
       "0  In what way do idea strings transmit tesion fo...        2058         290  \n",
       "1                    What was Tesla's device called?         181         202  \n",
       "2  What is the example of another problem charact...         282         581  \n",
       "3  What are EU Regulations essentially the same a...         759        1349  \n",
       "4  What theologian differed in views about the so...         412         188  \n",
       "5  What type of physics model did Einstein fail t...        2045        1468  \n",
       "6    Under what condition is an element irreducible?        1758         589  \n",
       "7  Of what does the covalent double bond result f...         630         317  \n",
       "8           What school model is Sweden notable for?        1343          53  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "squad_dataset.raw_test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section contains common variables and functions to be used when training all the subsequent models$\\dots$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell is tasked to load the default training parameters, such as the batch size, the logging frequency, where model checkpoints should be saved and more$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:17:10.946570Z",
     "iopub.status.busy": "2021-02-01T08:17:10.946070Z",
     "iopub.status.idle": "2021-02-01T08:17:11.010677Z",
     "shell.execute_reply": "2021-02-01T08:17:11.009425Z",
     "shell.execute_reply.started": "2021-02-01T08:17:10.946522Z"
    }
   },
   "outputs": [],
   "source": [
    "TRAINER_ARGS = utils.get_default_trainer_args()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section contains blocks of code that are used to train question answering models which are based on recurrent networks (`LSTM`s in our case). The models that we implemented are the following:\n",
    "- Baseline (a recurrent encoder with a naive version of attention)\n",
    "- BiDAF (Bi-Directional Attention Flow)\n",
    "\n",
    "Check the corresponding section to have a high-level view of each model$\\dots$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to load an embedding matrix using the `Gensim` API and use the corresponding matrix as the weight block of an `nn.Embedding` `PyTorch` module$\\dots$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we have to define one token for padding values. Then, OOV words are handled by a single unknown token, which is estimated as the mean of all the embedding vectors (if this mean vector is already present in the model, then a random embedding with suitable ranges is computed)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List of available embedding models (see [here](https://github.com/RaRe-Technologies/gensim-data)):\n",
    "- FastText: \n",
    "    - _fasttext-wiki-news-subwords_ (dimensions: 300)\n",
    "- GloVe:\n",
    "    - _glove-twitter_ (dimensions: 25. 50, 100, 200)\n",
    "    - _glove-wiki-gigaword_ (dimensions: 50, 100, 200, 300)\n",
    "- Word2Vec:\n",
    "    - _word2vec-google-news_ (dimensions: 300)\n",
    "    - _word2vec-ruscorpora_ (dimensions: 300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: The following cell could take a while (depending on the embedding dimension), since embedding models are pretty large$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:17:24.487486Z",
     "iopub.status.busy": "2021-02-01T08:17:24.487066Z",
     "iopub.status.idle": "2021-02-01T08:18:57.939122Z",
     "shell.execute_reply": "2021-02-01T08:18:57.937646Z",
     "shell.execute_reply.started": "2021-02-01T08:17:24.487440Z"
    }
   },
   "outputs": [],
   "source": [
    "embedding_model, vocab = utils.load_embedding_model(\n",
    "    config.EMBEDDING_MODEL_NAME,\n",
    "    embedding_dimension=config.EMBEDDING_DIMENSION,\n",
    "    unk_token=config.UNK_TOKEN,\n",
    "    pad_token=config.PAD_TOKEN,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell is tasked to load the embedding model into a `PyTorch` `nn.Embedding` layer, with frozen weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:18:59.928442Z",
     "iopub.status.busy": "2021-02-01T08:18:59.928120Z",
     "iopub.status.idle": "2021-02-01T08:19:01.215958Z",
     "shell.execute_reply": "2021-02-01T08:19:01.214661Z",
     "shell.execute_reply.started": "2021-02-01T08:18:59.928399Z"
    }
   },
   "outputs": [],
   "source": [
    "embedding_layer = layer_utils.get_embedding_module(\n",
    "    embedding_model, pad_id=vocab[config.PAD_TOKEN]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SquadDataManager` class acts as both a data collator (i.e. it brings together multiple examples in the dataset with the help of `PyTorch`'s `DataLoader`s) and a tokenizer. In particular, tokenization happens on the fly at the batch level, thus enabling us to perform dynamic padding (based on the longest sequence in a batch) and avoiding the pre-tokenization overhead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizer that we are using for recurrent modules splits words by whitespaces and punctuations, removes accents and applies a lowercasing function to all the tokens. Moreover, questions are padded (not truncated), while contexts are truncated to a maximum number of tokens and padded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:19:03.047069Z",
     "iopub.status.busy": "2021-02-01T08:19:03.046740Z",
     "iopub.status.idle": "2021-02-01T08:19:06.406582Z",
     "shell.execute_reply": "2021-02-01T08:19:06.404587Z",
     "shell.execute_reply.started": "2021-02-01T08:19:03.047026Z"
    }
   },
   "outputs": [],
   "source": [
    "recurrent_tokenizer = tokenizer.get_recurrent_tokenizer(\n",
    "    vocab,\n",
    "    config.MAX_CONTEXT_TOKENS,\n",
    "    config.UNK_TOKEN,\n",
    "    config.PAD_TOKEN,\n",
    "    device=DEVICE,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SquadDataManager` class also acts as a pre-processor, with the following steps:\n",
    "- Removes rows that contain wrong answers (e.g. answers that do not start and end at word boundaries)\n",
    "- Removes rows that contain answers that would be lost due to tokenization (truncation in particular)\n",
    "- Groups answers to the same question and context pair into a single row (thus producing lists in the `answer`, `answer_start` and `answer_end` columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:20:01.101984Z",
     "iopub.status.busy": "2021-02-01T08:20:01.101663Z",
     "iopub.status.idle": "2021-02-01T08:20:01.308673Z",
     "shell.execute_reply": "2021-02-01T08:20:01.307229Z",
     "shell.execute_reply.started": "2021-02-01T08:20:01.101941Z"
    }
   },
   "outputs": [],
   "source": [
    "recurrent_dm = dataset.SquadDataManager(\n",
    "    squad_dataset, recurrent_tokenizer, val_split=config.VAL_SPLIT, device=DEVICE\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last task assigned to the `SquadDataManager` class is that of train/validation splitting, with the given ratio ($80\\%$ for the training set by default)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the final training dataset$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:20:01.311264Z",
     "iopub.status.busy": "2021-02-01T08:20:01.310938Z",
     "iopub.status.idle": "2021-02-01T08:20:01.461295Z",
     "shell.execute_reply": "2021-02-01T08:20:01.460041Z",
     "shell.execute_reply.started": "2021-02-01T08:20:01.311221Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>question</th>\n",
       "      <th>title</th>\n",
       "      <th>context_id</th>\n",
       "      <th>context</th>\n",
       "      <th>answer</th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56cdd28562d2951400fa68bd</td>\n",
       "      <td>Who does M fight with?</td>\n",
       "      <td>Spectre_(2015_film)</td>\n",
       "      <td>470</td>\n",
       "      <td>Bond and Swann return to London where they mee...</td>\n",
       "      <td>[C]</td>\n",
       "      <td>[105]</td>\n",
       "      <td>[106]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>56cee5a1aab44d1400b88c24</td>\n",
       "      <td>In 1860, approximately how many people of Iris...</td>\n",
       "      <td>New_York_City</td>\n",
       "      <td>605</td>\n",
       "      <td>The Great Irish Famine brought a large influx ...</td>\n",
       "      <td>[200,000]</td>\n",
       "      <td>[72]</td>\n",
       "      <td>[79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>56cf83a7234ae51400d9bde3</td>\n",
       "      <td>Where was Donda West's funeral?</td>\n",
       "      <td>Kanye_West</td>\n",
       "      <td>1083</td>\n",
       "      <td>The funeral and burial for Donda West was held...</td>\n",
       "      <td>[Oklahoma City]</td>\n",
       "      <td>[50]</td>\n",
       "      <td>[63]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56d8d4e7bfea0914004b7728</td>\n",
       "      <td>What is the Olympic Torch made from?</td>\n",
       "      <td>2008_Summer_Olympics_torch_relay</td>\n",
       "      <td>1456</td>\n",
       "      <td>The Olympic Torch is based on traditional scro...</td>\n",
       "      <td>[aluminum.]</td>\n",
       "      <td>[128]</td>\n",
       "      <td>[137]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56de4d9ecffd8e1900b4b7e2</td>\n",
       "      <td>What year was the Banská Akadémia founded?</td>\n",
       "      <td>Institute_of_technology</td>\n",
       "      <td>1860</td>\n",
       "      <td>The world's first institution of technology or...</td>\n",
       "      <td>[1735]</td>\n",
       "      <td>[167]</td>\n",
       "      <td>[171]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>56e161bfcd28a01900c67849</td>\n",
       "      <td>What was the name of Bostons first baseball team?</td>\n",
       "      <td>Boston</td>\n",
       "      <td>3325</td>\n",
       "      <td>The Boston Red Sox, a founding member of the A...</td>\n",
       "      <td>[Red Stockings]</td>\n",
       "      <td>[798]</td>\n",
       "      <td>[811]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>56e1c12fe3433e140042312a</td>\n",
       "      <td>What TV station is the main public service bro...</td>\n",
       "      <td>Communications_in_Somalia</td>\n",
       "      <td>3418</td>\n",
       "      <td>The Mogadishu-based Somali National Television...</td>\n",
       "      <td>[Somali National Television]</td>\n",
       "      <td>[20]</td>\n",
       "      <td>[46]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>56e79e2e00c9c71400d773d5</td>\n",
       "      <td>Who convinced Sun Quan to make Nanjing his cap...</td>\n",
       "      <td>Nanjing</td>\n",
       "      <td>3759</td>\n",
       "      <td>Surrounded by the Yangtze River and mountains,...</td>\n",
       "      <td>[Liu Bei]</td>\n",
       "      <td>[389]</td>\n",
       "      <td>[396]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5709b2a2200fba1400368279</td>\n",
       "      <td>When did Houston begin to regain its dependenc...</td>\n",
       "      <td>Houston</td>\n",
       "      <td>5914</td>\n",
       "      <td>One wave of the population boom ended abruptly...</td>\n",
       "      <td>[2000s]</td>\n",
       "      <td>[571]</td>\n",
       "      <td>[576]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>570d2a80b3d812140066d4d3</td>\n",
       "      <td>Which city was the home of GE's first headquar...</td>\n",
       "      <td>General_Electric</td>\n",
       "      <td>7281</td>\n",
       "      <td>At about the same time, Charles Coffin, leadin...</td>\n",
       "      <td>[Schenectady]</td>\n",
       "      <td>[490]</td>\n",
       "      <td>[501]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>570d3f2ffed7b91900d45d98</td>\n",
       "      <td>What did the Observer Corps observe and report...</td>\n",
       "      <td>Anti-aircraft_warfare</td>\n",
       "      <td>7140</td>\n",
       "      <td>From the early 1930s eight countries developed...</td>\n",
       "      <td>[hostile aircraft flying over Britain]</td>\n",
       "      <td>[349]</td>\n",
       "      <td>[385]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>570e1a2a0dc6ce1900204dbf</td>\n",
       "      <td>How many species of fungi have been found on A...</td>\n",
       "      <td>Antarctica</td>\n",
       "      <td>6902</td>\n",
       "      <td>About 1150 species of fungi have been recorded...</td>\n",
       "      <td>[1150]</td>\n",
       "      <td>[6]</td>\n",
       "      <td>[10]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>570fa4675ab6b81900390f5f</td>\n",
       "      <td>What happens if the concepts are measured on t...</td>\n",
       "      <td>Sexual_orientation</td>\n",
       "      <td>7632</td>\n",
       "      <td>A third concern with the Kinsey scale is that ...</td>\n",
       "      <td>[they act as tradeoffs such, whereby to be mor...</td>\n",
       "      <td>[455]</td>\n",
       "      <td>[554]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>571af86f32177014007ea003</td>\n",
       "      <td>Who is responsible for approving drugs in the ...</td>\n",
       "      <td>Pharmaceutical_industry</td>\n",
       "      <td>8064</td>\n",
       "      <td>In the UK, the Medicines and Healthcare Produc...</td>\n",
       "      <td>[Medicines and Healthcare Products Regulatory ...</td>\n",
       "      <td>[15]</td>\n",
       "      <td>[66]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5725c70dec44d21400f3d540</td>\n",
       "      <td>Which part of the incandescent light bulb is h...</td>\n",
       "      <td>Incandescent_light_bulb</td>\n",
       "      <td>8805</td>\n",
       "      <td>An incandescent light bulb, incandescent lamp ...</td>\n",
       "      <td>[wire filament]</td>\n",
       "      <td>[102]</td>\n",
       "      <td>[115]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>57261a26ec44d21400f3d8dd</td>\n",
       "      <td>What is one of the names the Greeks call their...</td>\n",
       "      <td>Greece</td>\n",
       "      <td>9096</td>\n",
       "      <td>The names for the nation of Greece and the Gre...</td>\n",
       "      <td>[Hellas]</td>\n",
       "      <td>[164]</td>\n",
       "      <td>[170]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>572674a05951b619008f7319</td>\n",
       "      <td>What is another speed that can also be reporte...</td>\n",
       "      <td>Film_speed</td>\n",
       "      <td>9354</td>\n",
       "      <td>The standard specifies how speed ratings shoul...</td>\n",
       "      <td>[SOS-based speed]</td>\n",
       "      <td>[793]</td>\n",
       "      <td>[808]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>57268b4d5951b619008f7640</td>\n",
       "      <td>What does a DCT do?</td>\n",
       "      <td>Data_compression</td>\n",
       "      <td>10310</td>\n",
       "      <td>Today, nearly all commonly used video compress...</td>\n",
       "      <td>[spatial redundancy reduction]</td>\n",
       "      <td>[158]</td>\n",
       "      <td>[186]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>5726ef98708984140094d66e</td>\n",
       "      <td>What conferences became a requirement after Va...</td>\n",
       "      <td>Pope_Paul_VI</td>\n",
       "      <td>10862</td>\n",
       "      <td>Some critiqued Paul VI's decision; the newly c...</td>\n",
       "      <td>[National Bishop Conferences]</td>\n",
       "      <td>[347]</td>\n",
       "      <td>[374]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>5726fbec708984140094d7ad</td>\n",
       "      <td>Who named Madonna the Greatest Music Video sta...</td>\n",
       "      <td>Madonna_(entertainer)</td>\n",
       "      <td>10034</td>\n",
       "      <td>Academics noted that with her videos, Madonna ...</td>\n",
       "      <td>[MTV]</td>\n",
       "      <td>[483]</td>\n",
       "      <td>[486]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>57278aa6f1498d1400e8fb65</td>\n",
       "      <td>After losing the battle of Guilford Courthouse...</td>\n",
       "      <td>North_Carolina</td>\n",
       "      <td>12034</td>\n",
       "      <td>In the Battle of Cowan's Ford, Cornwallis met ...</td>\n",
       "      <td>[Virginia coastline]</td>\n",
       "      <td>[668]</td>\n",
       "      <td>[686]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>5727b0ccff5b5019007d92a9</td>\n",
       "      <td>How many companies developed USB's?</td>\n",
       "      <td>USB</td>\n",
       "      <td>12767</td>\n",
       "      <td>A group of seven companies began the developme...</td>\n",
       "      <td>[seven]</td>\n",
       "      <td>[11]</td>\n",
       "      <td>[16]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>5727b8b3ff5b5019007d936a</td>\n",
       "      <td>How does Darwin theorize that instincts have e...</td>\n",
       "      <td>On_the_Origin_of_Species</td>\n",
       "      <td>12124</td>\n",
       "      <td>Chapter VII (of the first edition) addresses t...</td>\n",
       "      <td>[He thought it reasonable that species with an...</td>\n",
       "      <td>[400]</td>\n",
       "      <td>[515]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>5727be06ff5b5019007d93fd</td>\n",
       "      <td>With whom does the primary responsibility for ...</td>\n",
       "      <td>Eton_College</td>\n",
       "      <td>12592</td>\n",
       "      <td>The primary responsibility for a boy's studies...</td>\n",
       "      <td>[House Master]</td>\n",
       "      <td>[61]</td>\n",
       "      <td>[73]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>572843ce4b864d190016485c</td>\n",
       "      <td>What was the purpose of top secret ICBM commit...</td>\n",
       "      <td>John_von_Neumann</td>\n",
       "      <td>11497</td>\n",
       "      <td>Shortly before his death, when he was already ...</td>\n",
       "      <td>[decide on the feasibility of building an ICBM...</td>\n",
       "      <td>[194]</td>\n",
       "      <td>[284]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>572ab10734ae481900deac33</td>\n",
       "      <td>How many engineering colleges are in Rajasthan?</td>\n",
       "      <td>Rajasthan</td>\n",
       "      <td>15059</td>\n",
       "      <td>In Rajasthan, Jodhpur and Kota are two major e...</td>\n",
       "      <td>[41 engineering colleges]</td>\n",
       "      <td>[742]</td>\n",
       "      <td>[765]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>572b63a8be1ee31400cb834d</td>\n",
       "      <td>What is Guam responsible for when goods both c...</td>\n",
       "      <td>Guam</td>\n",
       "      <td>15089</td>\n",
       "      <td>Guam is served by the Antonio B. Won Pat Inter...</td>\n",
       "      <td>[operating its own customs and quarantine agency]</td>\n",
       "      <td>[197]</td>\n",
       "      <td>[244]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>572f781404bcaa1900d769b8</td>\n",
       "      <td>When was ZE Records founded?</td>\n",
       "      <td>Post-punk</td>\n",
       "      <td>15248</td>\n",
       "      <td>Also emerging during this period was New York'...</td>\n",
       "      <td>[1978]</td>\n",
       "      <td>[668]</td>\n",
       "      <td>[672]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>572fc3ed04bcaa1900d76cb5</td>\n",
       "      <td>What drove Prussia away from renewing its alli...</td>\n",
       "      <td>Seven_Years%27_War</td>\n",
       "      <td>15338</td>\n",
       "      <td>The war had also brought to an end the \"Old Sy...</td>\n",
       "      <td>[Prussians were angered by what they considere...</td>\n",
       "      <td>[452]</td>\n",
       "      <td>[525]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>5730bb058ab72b1400f9c72c</td>\n",
       "      <td>Where were the use of advanced materials and t...</td>\n",
       "      <td>Sumer</td>\n",
       "      <td>17505</td>\n",
       "      <td>The most impressive and famous of Sumerian bui...</td>\n",
       "      <td>[Sumerian temples and palaces]</td>\n",
       "      <td>[421]</td>\n",
       "      <td>[449]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>5730e7c0b54a4f140068ccd2</td>\n",
       "      <td>When did the US buy Alaska?</td>\n",
       "      <td>Russian_language</td>\n",
       "      <td>17646</td>\n",
       "      <td>The language was first introduced in North Ame...</td>\n",
       "      <td>[1867]</td>\n",
       "      <td>[210]</td>\n",
       "      <td>[214]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>57324644e17f3d14004227b3</td>\n",
       "      <td>What kind of music does Sayat Nova play?</td>\n",
       "      <td>Armenians</td>\n",
       "      <td>18452</td>\n",
       "      <td>Instruments like the duduk, the dhol, the zurn...</td>\n",
       "      <td>[Armenian folk music]</td>\n",
       "      <td>[188]</td>\n",
       "      <td>[207]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>57326b2be99e3014001e678a</td>\n",
       "      <td>What country did Eisenhower believe communists...</td>\n",
       "      <td>Dwight_D._Eisenhower</td>\n",
       "      <td>18556</td>\n",
       "      <td>In 1954, Eisenhower articulated the domino the...</td>\n",
       "      <td>[Mexico]</td>\n",
       "      <td>[399]</td>\n",
       "      <td>[405]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>5733acebd058e614000b600b</td>\n",
       "      <td>When did Tajiks start being part of the Soviet...</td>\n",
       "      <td>Tajikistan</td>\n",
       "      <td>846</td>\n",
       "      <td>Tajiks began to be conscripted into the Soviet...</td>\n",
       "      <td>[1939]</td>\n",
       "      <td>[55]</td>\n",
       "      <td>[59]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 question_id  \\\n",
       "0   56cdd28562d2951400fa68bd   \n",
       "1   56cee5a1aab44d1400b88c24   \n",
       "2   56cf83a7234ae51400d9bde3   \n",
       "3   56d8d4e7bfea0914004b7728   \n",
       "4   56de4d9ecffd8e1900b4b7e2   \n",
       "5   56e161bfcd28a01900c67849   \n",
       "6   56e1c12fe3433e140042312a   \n",
       "7   56e79e2e00c9c71400d773d5   \n",
       "8   5709b2a2200fba1400368279   \n",
       "9   570d2a80b3d812140066d4d3   \n",
       "10  570d3f2ffed7b91900d45d98   \n",
       "11  570e1a2a0dc6ce1900204dbf   \n",
       "12  570fa4675ab6b81900390f5f   \n",
       "13  571af86f32177014007ea003   \n",
       "14  5725c70dec44d21400f3d540   \n",
       "15  57261a26ec44d21400f3d8dd   \n",
       "16  572674a05951b619008f7319   \n",
       "17  57268b4d5951b619008f7640   \n",
       "18  5726ef98708984140094d66e   \n",
       "19  5726fbec708984140094d7ad   \n",
       "20  57278aa6f1498d1400e8fb65   \n",
       "21  5727b0ccff5b5019007d92a9   \n",
       "22  5727b8b3ff5b5019007d936a   \n",
       "23  5727be06ff5b5019007d93fd   \n",
       "24  572843ce4b864d190016485c   \n",
       "25  572ab10734ae481900deac33   \n",
       "26  572b63a8be1ee31400cb834d   \n",
       "27  572f781404bcaa1900d769b8   \n",
       "28  572fc3ed04bcaa1900d76cb5   \n",
       "29  5730bb058ab72b1400f9c72c   \n",
       "30  5730e7c0b54a4f140068ccd2   \n",
       "31  57324644e17f3d14004227b3   \n",
       "32  57326b2be99e3014001e678a   \n",
       "33  5733acebd058e614000b600b   \n",
       "\n",
       "                                             question  \\\n",
       "0                              Who does M fight with?   \n",
       "1   In 1860, approximately how many people of Iris...   \n",
       "2                     Where was Donda West's funeral?   \n",
       "3                What is the Olympic Torch made from?   \n",
       "4          What year was the Banská Akadémia founded?   \n",
       "5   What was the name of Bostons first baseball team?   \n",
       "6   What TV station is the main public service bro...   \n",
       "7   Who convinced Sun Quan to make Nanjing his cap...   \n",
       "8   When did Houston begin to regain its dependenc...   \n",
       "9   Which city was the home of GE's first headquar...   \n",
       "10  What did the Observer Corps observe and report...   \n",
       "11  How many species of fungi have been found on A...   \n",
       "12  What happens if the concepts are measured on t...   \n",
       "13  Who is responsible for approving drugs in the ...   \n",
       "14  Which part of the incandescent light bulb is h...   \n",
       "15  What is one of the names the Greeks call their...   \n",
       "16  What is another speed that can also be reporte...   \n",
       "17                                What does a DCT do?   \n",
       "18  What conferences became a requirement after Va...   \n",
       "19  Who named Madonna the Greatest Music Video sta...   \n",
       "20  After losing the battle of Guilford Courthouse...   \n",
       "21                How many companies developed USB's?   \n",
       "22  How does Darwin theorize that instincts have e...   \n",
       "23  With whom does the primary responsibility for ...   \n",
       "24  What was the purpose of top secret ICBM commit...   \n",
       "25    How many engineering colleges are in Rajasthan?   \n",
       "26  What is Guam responsible for when goods both c...   \n",
       "27                       When was ZE Records founded?   \n",
       "28  What drove Prussia away from renewing its alli...   \n",
       "29  Where were the use of advanced materials and t...   \n",
       "30                        When did the US buy Alaska?   \n",
       "31           What kind of music does Sayat Nova play?   \n",
       "32  What country did Eisenhower believe communists...   \n",
       "33  When did Tajiks start being part of the Soviet...   \n",
       "\n",
       "                               title  context_id  \\\n",
       "0                Spectre_(2015_film)         470   \n",
       "1                      New_York_City         605   \n",
       "2                         Kanye_West        1083   \n",
       "3   2008_Summer_Olympics_torch_relay        1456   \n",
       "4            Institute_of_technology        1860   \n",
       "5                             Boston        3325   \n",
       "6          Communications_in_Somalia        3418   \n",
       "7                            Nanjing        3759   \n",
       "8                            Houston        5914   \n",
       "9                   General_Electric        7281   \n",
       "10             Anti-aircraft_warfare        7140   \n",
       "11                        Antarctica        6902   \n",
       "12                Sexual_orientation        7632   \n",
       "13           Pharmaceutical_industry        8064   \n",
       "14           Incandescent_light_bulb        8805   \n",
       "15                            Greece        9096   \n",
       "16                        Film_speed        9354   \n",
       "17                  Data_compression       10310   \n",
       "18                      Pope_Paul_VI       10862   \n",
       "19             Madonna_(entertainer)       10034   \n",
       "20                    North_Carolina       12034   \n",
       "21                               USB       12767   \n",
       "22          On_the_Origin_of_Species       12124   \n",
       "23                      Eton_College       12592   \n",
       "24                  John_von_Neumann       11497   \n",
       "25                         Rajasthan       15059   \n",
       "26                              Guam       15089   \n",
       "27                         Post-punk       15248   \n",
       "28                Seven_Years%27_War       15338   \n",
       "29                             Sumer       17505   \n",
       "30                  Russian_language       17646   \n",
       "31                         Armenians       18452   \n",
       "32              Dwight_D._Eisenhower       18556   \n",
       "33                        Tajikistan         846   \n",
       "\n",
       "                                              context  \\\n",
       "0   Bond and Swann return to London where they mee...   \n",
       "1   The Great Irish Famine brought a large influx ...   \n",
       "2   The funeral and burial for Donda West was held...   \n",
       "3   The Olympic Torch is based on traditional scro...   \n",
       "4   The world's first institution of technology or...   \n",
       "5   The Boston Red Sox, a founding member of the A...   \n",
       "6   The Mogadishu-based Somali National Television...   \n",
       "7   Surrounded by the Yangtze River and mountains,...   \n",
       "8   One wave of the population boom ended abruptly...   \n",
       "9   At about the same time, Charles Coffin, leadin...   \n",
       "10  From the early 1930s eight countries developed...   \n",
       "11  About 1150 species of fungi have been recorded...   \n",
       "12  A third concern with the Kinsey scale is that ...   \n",
       "13  In the UK, the Medicines and Healthcare Produc...   \n",
       "14  An incandescent light bulb, incandescent lamp ...   \n",
       "15  The names for the nation of Greece and the Gre...   \n",
       "16  The standard specifies how speed ratings shoul...   \n",
       "17  Today, nearly all commonly used video compress...   \n",
       "18  Some critiqued Paul VI's decision; the newly c...   \n",
       "19  Academics noted that with her videos, Madonna ...   \n",
       "20  In the Battle of Cowan's Ford, Cornwallis met ...   \n",
       "21  A group of seven companies began the developme...   \n",
       "22  Chapter VII (of the first edition) addresses t...   \n",
       "23  The primary responsibility for a boy's studies...   \n",
       "24  Shortly before his death, when he was already ...   \n",
       "25  In Rajasthan, Jodhpur and Kota are two major e...   \n",
       "26  Guam is served by the Antonio B. Won Pat Inter...   \n",
       "27  Also emerging during this period was New York'...   \n",
       "28  The war had also brought to an end the \"Old Sy...   \n",
       "29  The most impressive and famous of Sumerian bui...   \n",
       "30  The language was first introduced in North Ame...   \n",
       "31  Instruments like the duduk, the dhol, the zurn...   \n",
       "32  In 1954, Eisenhower articulated the domino the...   \n",
       "33  Tajiks began to be conscripted into the Soviet...   \n",
       "\n",
       "                                               answer answer_start answer_end  \n",
       "0                                                 [C]        [105]      [106]  \n",
       "1                                           [200,000]         [72]       [79]  \n",
       "2                                     [Oklahoma City]         [50]       [63]  \n",
       "3                                         [aluminum.]        [128]      [137]  \n",
       "4                                              [1735]        [167]      [171]  \n",
       "5                                     [Red Stockings]        [798]      [811]  \n",
       "6                        [Somali National Television]         [20]       [46]  \n",
       "7                                           [Liu Bei]        [389]      [396]  \n",
       "8                                             [2000s]        [571]      [576]  \n",
       "9                                       [Schenectady]        [490]      [501]  \n",
       "10             [hostile aircraft flying over Britain]        [349]      [385]  \n",
       "11                                             [1150]          [6]       [10]  \n",
       "12  [they act as tradeoffs such, whereby to be mor...        [455]      [554]  \n",
       "13  [Medicines and Healthcare Products Regulatory ...         [15]       [66]  \n",
       "14                                    [wire filament]        [102]      [115]  \n",
       "15                                           [Hellas]        [164]      [170]  \n",
       "16                                  [SOS-based speed]        [793]      [808]  \n",
       "17                     [spatial redundancy reduction]        [158]      [186]  \n",
       "18                      [National Bishop Conferences]        [347]      [374]  \n",
       "19                                              [MTV]        [483]      [486]  \n",
       "20                               [Virginia coastline]        [668]      [686]  \n",
       "21                                            [seven]         [11]       [16]  \n",
       "22  [He thought it reasonable that species with an...        [400]      [515]  \n",
       "23                                     [House Master]         [61]       [73]  \n",
       "24  [decide on the feasibility of building an ICBM...        [194]      [284]  \n",
       "25                          [41 engineering colleges]        [742]      [765]  \n",
       "26  [operating its own customs and quarantine agency]        [197]      [244]  \n",
       "27                                             [1978]        [668]      [672]  \n",
       "28  [Prussians were angered by what they considere...        [452]      [525]  \n",
       "29                     [Sumerian temples and palaces]        [421]      [449]  \n",
       "30                                             [1867]        [210]      [214]  \n",
       "31                              [Armenian folk music]        [188]      [207]  \n",
       "32                                           [Mexico]        [399]      [405]  \n",
       "33                                             [1939]         [55]       [59]  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recurrent_dm.train_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the final validation dataset$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:20:05.329187Z",
     "iopub.status.busy": "2021-02-01T08:20:05.328862Z",
     "iopub.status.idle": "2021-02-01T08:20:05.423735Z",
     "shell.execute_reply": "2021-02-01T08:20:05.422633Z",
     "shell.execute_reply.started": "2021-02-01T08:20:05.329144Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>question</th>\n",
       "      <th>title</th>\n",
       "      <th>context_id</th>\n",
       "      <th>context</th>\n",
       "      <th>answer</th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56ccf53362d2951400fa64fd</td>\n",
       "      <td>When did Ayurbarwada Buyantu Khan reign?</td>\n",
       "      <td>Sino-Tibetan_relations_during_the_Ming_dynasty</td>\n",
       "      <td>319</td>\n",
       "      <td>Nevertheless, the ethno-geographic caste hiera...</td>\n",
       "      <td>[1311–1320]</td>\n",
       "      <td>[447]</td>\n",
       "      <td>[456]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>56cd480b62d2951400fa6510</td>\n",
       "      <td>How many households were the offices of Qianhu...</td>\n",
       "      <td>Sino-Tibetan_relations_during_the_Ming_dynasty</td>\n",
       "      <td>321</td>\n",
       "      <td>Chen Qingying, Professor of History and Direct...</td>\n",
       "      <td>[1,000 households]</td>\n",
       "      <td>[690]</td>\n",
       "      <td>[706]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5726dde0f1498d1400e8edf7</td>\n",
       "      <td>Why was Professor David Graeber retired during...</td>\n",
       "      <td>Yale_University</td>\n",
       "      <td>10500</td>\n",
       "      <td>Yale has a history of difficult and prolonged ...</td>\n",
       "      <td>[he came to the defense of a student who was i...</td>\n",
       "      <td>[568]</td>\n",
       "      <td>[644]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>572755d8708984140094dc5a</td>\n",
       "      <td>How big is the Matthaei botanical garden?</td>\n",
       "      <td>Ann_Arbor,_Michigan</td>\n",
       "      <td>10568</td>\n",
       "      <td>Ann Arbor's \"Tree Town\" nickname stems from th...</td>\n",
       "      <td>[300 acres]</td>\n",
       "      <td>[967]</td>\n",
       "      <td>[976]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>572781a5f1498d1400e8fa1f</td>\n",
       "      <td>Who is elected every even numbered year?</td>\n",
       "      <td>Ann_Arbor,_Michigan</td>\n",
       "      <td>10585</td>\n",
       "      <td>Ann Arbor has a council-manager form of govern...</td>\n",
       "      <td>[mayor]</td>\n",
       "      <td>[192]</td>\n",
       "      <td>[197]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>57283caa3acd2414000df780</td>\n",
       "      <td>How many civilians died in the attack?</td>\n",
       "      <td>Dissolution_of_the_Soviet_Union</td>\n",
       "      <td>12205</td>\n",
       "      <td>On January 13, 1991, Soviet troops, along with...</td>\n",
       "      <td>[Fourteen]</td>\n",
       "      <td>[158]</td>\n",
       "      <td>[166]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>572a301c3f37b31900478791</td>\n",
       "      <td>What did Europeans refer to the Ottoman empire...</td>\n",
       "      <td>Ottoman_Empire</td>\n",
       "      <td>14726</td>\n",
       "      <td>The Serbian revolution (1804–1815) marked the ...</td>\n",
       "      <td>[the \"sick man\"]</td>\n",
       "      <td>[588]</td>\n",
       "      <td>[602]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>572f785e04bcaa1900d769c0</td>\n",
       "      <td>Why did the Luftwaffe bomb the RAF Fighter Com...</td>\n",
       "      <td>The_Blitz</td>\n",
       "      <td>15807</td>\n",
       "      <td>Although not specifically prepared to conduct ...</td>\n",
       "      <td>[to gain air superiority]</td>\n",
       "      <td>[228]</td>\n",
       "      <td>[251]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>572fabd004bcaa1900d76bac</td>\n",
       "      <td>What year did the government start giving out ...</td>\n",
       "      <td>The_Blitz</td>\n",
       "      <td>15820</td>\n",
       "      <td>Communal shelters never housed more than one s...</td>\n",
       "      <td>[1941]</td>\n",
       "      <td>[457]</td>\n",
       "      <td>[461]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                question_id  \\\n",
       "0  56ccf53362d2951400fa64fd   \n",
       "1  56cd480b62d2951400fa6510   \n",
       "2  5726dde0f1498d1400e8edf7   \n",
       "3  572755d8708984140094dc5a   \n",
       "4  572781a5f1498d1400e8fa1f   \n",
       "5  57283caa3acd2414000df780   \n",
       "6  572a301c3f37b31900478791   \n",
       "7  572f785e04bcaa1900d769c0   \n",
       "8  572fabd004bcaa1900d76bac   \n",
       "\n",
       "                                            question  \\\n",
       "0           When did Ayurbarwada Buyantu Khan reign?   \n",
       "1  How many households were the offices of Qianhu...   \n",
       "2  Why was Professor David Graeber retired during...   \n",
       "3          How big is the Matthaei botanical garden?   \n",
       "4           Who is elected every even numbered year?   \n",
       "5             How many civilians died in the attack?   \n",
       "6  What did Europeans refer to the Ottoman empire...   \n",
       "7  Why did the Luftwaffe bomb the RAF Fighter Com...   \n",
       "8  What year did the government start giving out ...   \n",
       "\n",
       "                                            title  context_id  \\\n",
       "0  Sino-Tibetan_relations_during_the_Ming_dynasty         319   \n",
       "1  Sino-Tibetan_relations_during_the_Ming_dynasty         321   \n",
       "2                                 Yale_University       10500   \n",
       "3                             Ann_Arbor,_Michigan       10568   \n",
       "4                             Ann_Arbor,_Michigan       10585   \n",
       "5                 Dissolution_of_the_Soviet_Union       12205   \n",
       "6                                  Ottoman_Empire       14726   \n",
       "7                                       The_Blitz       15807   \n",
       "8                                       The_Blitz       15820   \n",
       "\n",
       "                                             context  \\\n",
       "0  Nevertheless, the ethno-geographic caste hiera...   \n",
       "1  Chen Qingying, Professor of History and Direct...   \n",
       "2  Yale has a history of difficult and prolonged ...   \n",
       "3  Ann Arbor's \"Tree Town\" nickname stems from th...   \n",
       "4  Ann Arbor has a council-manager form of govern...   \n",
       "5  On January 13, 1991, Soviet troops, along with...   \n",
       "6  The Serbian revolution (1804–1815) marked the ...   \n",
       "7  Although not specifically prepared to conduct ...   \n",
       "8  Communal shelters never housed more than one s...   \n",
       "\n",
       "                                              answer answer_start answer_end  \n",
       "0                                        [1311–1320]        [447]      [456]  \n",
       "1                                 [1,000 households]        [690]      [706]  \n",
       "2  [he came to the defense of a student who was i...        [568]      [644]  \n",
       "3                                        [300 acres]        [967]      [976]  \n",
       "4                                            [mayor]        [192]      [197]  \n",
       "5                                         [Fourteen]        [158]      [166]  \n",
       "6                                   [the \"sick man\"]        [588]      [602]  \n",
       "7                          [to gain air superiority]        [228]      [251]  \n",
       "8                                             [1941]        [457]      [461]  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recurrent_dm.val_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's do the same for the testing dataset$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:20:05.425815Z",
     "iopub.status.busy": "2021-02-01T08:20:05.425503Z",
     "iopub.status.idle": "2021-02-01T08:20:05.512431Z",
     "shell.execute_reply": "2021-02-01T08:20:05.511385Z",
     "shell.execute_reply.started": "2021-02-01T08:20:05.425775Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>question</th>\n",
       "      <th>title</th>\n",
       "      <th>context_id</th>\n",
       "      <th>context</th>\n",
       "      <th>answer</th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56e0ed557aa994140058e7dd</td>\n",
       "      <td>What was Tesla's device called?</td>\n",
       "      <td>Nikola_Tesla</td>\n",
       "      <td>181</td>\n",
       "      <td>Tesla also explained the principles of the rot...</td>\n",
       "      <td>[Egg of Columbus]</td>\n",
       "      <td>[187]</td>\n",
       "      <td>[202]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>56e1febfe3433e140042323a</td>\n",
       "      <td>What is the example of another problem charact...</td>\n",
       "      <td>Computational_complexity_theory</td>\n",
       "      <td>282</td>\n",
       "      <td>What intractability means in practice is open ...</td>\n",
       "      <td>[NP-complete Boolean satisfiability problem]</td>\n",
       "      <td>[539]</td>\n",
       "      <td>[581]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>56f884cba6d7ea1400e17708</td>\n",
       "      <td>What theologian differed in views about the so...</td>\n",
       "      <td>Martin_Luther</td>\n",
       "      <td>412</td>\n",
       "      <td>The Lutheran theologian Franz Pieper observed ...</td>\n",
       "      <td>[Gerhard]</td>\n",
       "      <td>[181]</td>\n",
       "      <td>[188]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>571c83f3dd7acb1400e4c0dc</td>\n",
       "      <td>Of what does the covalent double bond result f...</td>\n",
       "      <td>Oxygen</td>\n",
       "      <td>630</td>\n",
       "      <td>In this dioxygen, the two oxygen atoms are che...</td>\n",
       "      <td>[the filling of molecular orbitals formed from...</td>\n",
       "      <td>[221]</td>\n",
       "      <td>[317]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57269bb8708984140094cb98</td>\n",
       "      <td>What are EU Regulations essentially the same a...</td>\n",
       "      <td>European_Union_law</td>\n",
       "      <td>759</td>\n",
       "      <td>Although it is generally accepted that EU law ...</td>\n",
       "      <td>[Treaty provisions]</td>\n",
       "      <td>[1332]</td>\n",
       "      <td>[1349]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>572754dd708984140094dc3f</td>\n",
       "      <td>What school model is Sweden notable for?</td>\n",
       "      <td>Private_school</td>\n",
       "      <td>1343</td>\n",
       "      <td>In Sweden, pupils are free to choose a private...</td>\n",
       "      <td>[pupils are free to choose a private school]</td>\n",
       "      <td>[11]</td>\n",
       "      <td>[53]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>57299c2c6aef051400155024</td>\n",
       "      <td>Under what condition is an element irreducible?</td>\n",
       "      <td>Prime_number</td>\n",
       "      <td>1758</td>\n",
       "      <td>Prime numbers give rise to two more general co...</td>\n",
       "      <td>[not a unit and cannot be written as a product...</td>\n",
       "      <td>[503]</td>\n",
       "      <td>[589]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>5737821cc3c5551400e51f1c</td>\n",
       "      <td>What type of physics model did Einstein fail t...</td>\n",
       "      <td>Force</td>\n",
       "      <td>2045</td>\n",
       "      <td>The development of fundamental theories for fo...</td>\n",
       "      <td>[self-consistent unification]</td>\n",
       "      <td>[1441]</td>\n",
       "      <td>[1468]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>57379ed81c456719005744d7</td>\n",
       "      <td>In what way do idea strings transmit tesion fo...</td>\n",
       "      <td>Force</td>\n",
       "      <td>2058</td>\n",
       "      <td>Tension forces can be modeled using ideal stri...</td>\n",
       "      <td>[instantaneously in action-reaction pairs]</td>\n",
       "      <td>[250]</td>\n",
       "      <td>[290]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                question_id  \\\n",
       "0  56e0ed557aa994140058e7dd   \n",
       "1  56e1febfe3433e140042323a   \n",
       "2  56f884cba6d7ea1400e17708   \n",
       "3  571c83f3dd7acb1400e4c0dc   \n",
       "4  57269bb8708984140094cb98   \n",
       "5  572754dd708984140094dc3f   \n",
       "6  57299c2c6aef051400155024   \n",
       "7  5737821cc3c5551400e51f1c   \n",
       "8  57379ed81c456719005744d7   \n",
       "\n",
       "                                            question  \\\n",
       "0                    What was Tesla's device called?   \n",
       "1  What is the example of another problem charact...   \n",
       "2  What theologian differed in views about the so...   \n",
       "3  Of what does the covalent double bond result f...   \n",
       "4  What are EU Regulations essentially the same a...   \n",
       "5           What school model is Sweden notable for?   \n",
       "6    Under what condition is an element irreducible?   \n",
       "7  What type of physics model did Einstein fail t...   \n",
       "8  In what way do idea strings transmit tesion fo...   \n",
       "\n",
       "                             title  context_id  \\\n",
       "0                     Nikola_Tesla         181   \n",
       "1  Computational_complexity_theory         282   \n",
       "2                    Martin_Luther         412   \n",
       "3                           Oxygen         630   \n",
       "4               European_Union_law         759   \n",
       "5                   Private_school        1343   \n",
       "6                     Prime_number        1758   \n",
       "7                            Force        2045   \n",
       "8                            Force        2058   \n",
       "\n",
       "                                             context  \\\n",
       "0  Tesla also explained the principles of the rot...   \n",
       "1  What intractability means in practice is open ...   \n",
       "2  The Lutheran theologian Franz Pieper observed ...   \n",
       "3  In this dioxygen, the two oxygen atoms are che...   \n",
       "4  Although it is generally accepted that EU law ...   \n",
       "5  In Sweden, pupils are free to choose a private...   \n",
       "6  Prime numbers give rise to two more general co...   \n",
       "7  The development of fundamental theories for fo...   \n",
       "8  Tension forces can be modeled using ideal stri...   \n",
       "\n",
       "                                              answer answer_start answer_end  \n",
       "0                                  [Egg of Columbus]        [187]      [202]  \n",
       "1       [NP-complete Boolean satisfiability problem]        [539]      [581]  \n",
       "2                                          [Gerhard]        [181]      [188]  \n",
       "3  [the filling of molecular orbitals formed from...        [221]      [317]  \n",
       "4                                [Treaty provisions]       [1332]     [1349]  \n",
       "5       [pupils are free to choose a private school]         [11]       [53]  \n",
       "6  [not a unit and cannot be written as a product...        [503]      [589]  \n",
       "7                      [self-consistent unification]       [1441]     [1468]  \n",
       "8         [instantaneously in action-reaction pairs]        [250]      [290]  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recurrent_dm.test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The baseline model is composed by a single recurrent encoder, which is given both questions and contexts as two separate inputs. Then, all the hidden states of a single question are averaged together (over the embedding dimension) so as to obtain a single vector which should encode the semantic information of the question at the sentence level. This aggregated question vector is then element-wise multiplied to each context token latent representation, so as to perform some kind of query-aware context encoding. Finally, the query-aware context vectors are passed onto another recurrent module and used as inputs for the end token classifier, while the query-aware context vectors are directly used as input for the start token classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperparameters are carefully chosen by hand, considering the fact that the baseline model is pretty lightweight (in terms of FLOPS and parameters), so that we can afford using higher batch sizes. Moreover, the number of training epoch is set to a high value to understand if and when overfitting is observed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:20:23.166973Z",
     "iopub.status.busy": "2021-02-01T08:20:23.166527Z",
     "iopub.status.idle": "2021-02-01T08:20:23.235873Z",
     "shell.execute_reply": "2021-02-01T08:20:23.234658Z",
     "shell.execute_reply.started": "2021-02-01T08:20:23.166926Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_RUN_GROUP=baseline\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_RUN_GROUP=baseline\n",
    "baseline_run_name = utils.get_run_name()\n",
    "baseline_args = partial(\n",
    "    TRAINER_ARGS,\n",
    "    output_dir=f\"./checkpoints/{os.getenv('WANDB_RUN_GROUP')}/{baseline_run_name}\",\n",
    "    num_train_epochs=30,\n",
    "    per_device_train_batch_size=128,\n",
    "    per_device_eval_batch_size=128,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training and validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will use the splitted train dataset to perform training and the splitted validation dataset to observe the evolution of metrics during training, at the end of each epoch. To be clear, the validation set is not used to tune hyperparameters, but just as a reference to report results over unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:35:58.157392Z",
     "iopub.status.busy": "2021-02-01T08:35:58.156962Z",
     "iopub.status.idle": "2021-02-01T08:35:58.249904Z",
     "shell.execute_reply": "2021-02-01T08:35:58.248804Z",
     "shell.execute_reply.started": "2021-02-01T08:35:58.157344Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The baseline model has 245202 parameters\n"
     ]
    }
   ],
   "source": [
    "baseline_model = model.QABaselineModel(embedding_layer, device=DEVICE)\n",
    "print(f\"The baseline model has {baseline_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:36:01.172876Z",
     "iopub.status.busy": "2021-02-01T08:36:01.172481Z",
     "iopub.status.idle": "2021-02-01T08:36:01.251133Z",
     "shell.execute_reply": "2021-02-01T08:36:01.249996Z",
     "shell.execute_reply.started": "2021-02-01T08:36:01.172831Z"
    }
   },
   "outputs": [],
   "source": [
    "baseline_optimizer = optim.Adam(baseline_model.parameters(), lr=1e-3)\n",
    "baseline_lr_scheduler = transformers.get_constant_schedule(baseline_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-01T08:36:03.304369Z",
     "iopub.status.busy": "2021-02-01T08:36:03.304022Z",
     "iopub.status.idle": "2021-02-01T08:36:03.397194Z",
     "shell.execute_reply": "2021-02-01T08:36:03.396118Z",
     "shell.execute_reply.started": "2021-02-01T08:36:03.304325Z"
    }
   },
   "outputs": [],
   "source": [
    "baseline_trainer = training.SquadTrainer(\n",
    "    model=baseline_model,\n",
    "    args=baseline_args(run_name=baseline_run_name),\n",
    "    data_collator=recurrent_dm.tokenizer,\n",
    "    train_dataset=recurrent_dm.train_dataset,\n",
    "    eval_dataset=recurrent_dm.val_dataset,\n",
    "    optimizers=(baseline_optimizer, baseline_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to use the whole dataset for training, so only the training loss will be used as a metric. This step is used to boost generalization ability and to exploit all the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T09:58:34.597944Z",
     "iopub.status.busy": "2021-01-24T09:58:34.597582Z",
     "iopub.status.idle": "2021-01-24T09:58:34.676414Z",
     "shell.execute_reply": "2021-01-24T09:58:34.675144Z",
     "shell.execute_reply.started": "2021-01-24T09:58:34.597901Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The baseline model has 41000 parameters\n"
     ]
    }
   ],
   "source": [
    "baseline_model = model.QABaselineModel(embedding_layer, device=DEVICE)\n",
    "print(f\"The baseline model has {baseline_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_optimizer = optim.Adam(baseline_model.parameters(), lr=1e-3)\n",
    "baseline_lr_scheduler = transformers.get_constant_schedule(baseline_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T09:58:36.906188Z",
     "iopub.status.busy": "2021-01-24T09:58:36.905856Z",
     "iopub.status.idle": "2021-01-24T09:58:36.988918Z",
     "shell.execute_reply": "2021-01-24T09:58:36.987625Z",
     "shell.execute_reply.started": "2021-01-24T09:58:36.906146Z"
    }
   },
   "outputs": [],
   "source": [
    "baseline_trainer = training.SquadTrainer(\n",
    "    model=baseline_model,\n",
    "    args=baseline_args(run_name=f\"{baseline_run_name}-whole\", evaluation_strategy=\"no\"),\n",
    "    data_collator=recurrent_dm.tokenizer,\n",
    "    train_dataset=recurrent_dm.whole_dataset,\n",
    "    optimizers=(baseline_optimizer, baseline_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will use the test set to assess the generalization abilities of our model and observe final metrics$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_test_output = baseline_trainer.predict(recurrent_dm.test_dataset)\n",
    "baseline_test_output.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we are going to save a `JSON` file with the following schema:\n",
    "```json\n",
    "{\n",
    "    \"question_id\": \"textual answer\"\n",
    "    ...\n",
    "}\n",
    "```\n",
    "The `JSON` files contains textual answers to each question in the given test dataset. The output file can also be used with the `SQuAD` official evaluation script$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T10:04:58.240993Z",
     "iopub.status.busy": "2021-01-24T10:04:58.240640Z",
     "iopub.status.idle": "2021-01-24T10:05:16.055426Z",
     "shell.execute_reply": "2021-01-24T10:05:16.053922Z",
     "shell.execute_reply.started": "2021-01-24T10:04:58.240950Z"
    }
   },
   "outputs": [],
   "source": [
    "baseline_answers_path = \"results/answers/baseline.json\"\n",
    "utils.save_answers(baseline_answers_path, baseline_test_output.predictions[-1])\n",
    "wandb.save(baseline_answers_path);\n",
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BiDAF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-26T11:52:15.838690Z",
     "iopub.status.busy": "2021-01-26T11:52:15.838204Z",
     "iopub.status.idle": "2021-01-26T11:52:15.905398Z",
     "shell.execute_reply": "2021-01-26T11:52:15.903846Z",
     "shell.execute_reply.started": "2021-01-26T11:52:15.838644Z"
    }
   },
   "source": [
    "The Bi-Directional Attention Flow (BIDAF) network is a hierarchical\n",
    "multi-stage architecture for modeling the representations of the context paragraph at different levels\n",
    "of granularity. BIDAF includes character-level, word-level, and contextual embeddings,\n",
    "and uses bi-directional attention flow to obtain a query-aware context representation.\n",
    "Our attention mechanism offers following improvements to the previously popular attention paradigms. \n",
    "First, the attention layer is not used to summarize the context paragraph into a fixed-size vector. Instead, the\n",
    "attention is computed for every time step, and the attended vector at each time step, along with the\n",
    "representations from previous layers, is allowed to flow through to the subsequent modeling layer.\n",
    "This reduces the information loss caused by early summarization. Second, we use a memory-less\n",
    "attention mechanism. That is, while we iteratively compute attention through time, the attention at each time step is a function of only the query and the context paragraph at the current time step and does not directly depend on the attention at the previous time step.\n",
    "\n",
    "The abstract from the paper is the following:\n",
    "> Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query. Recently, attention mechanisms have been successfully extended to MC. Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention. In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization. Our experimental evaluations show that our model achieves the state-of-the-art results in Stanford Question Answering Dataset (SQuAD) and CNN/DailyMail cloze test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main difference of our BiDAF model w.r.t. the full model reported in the paper is that we decided to avoid using character embeddings, since ablation studies showed that it gives only marginal improvements$\\dots$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperparameters are taken directly from the BiDAF paper:\n",
    "- Epochs: $12$ (we also tried an higher number of epochs)\n",
    "- Batch size: $60$\n",
    "- Optimizer: Adadelta\n",
    "- Learning rate: $0.5$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T14:31:09.693611Z",
     "iopub.status.busy": "2021-01-24T14:31:09.693249Z",
     "iopub.status.idle": "2021-01-24T14:31:09.785595Z",
     "shell.execute_reply": "2021-01-24T14:31:09.784353Z",
     "shell.execute_reply.started": "2021-01-24T14:31:09.693567Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_RUN_GROUP=bidaf\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_RUN_GROUP=bidaf\n",
    "bidaf_run_name = utils.get_run_name()\n",
    "bidaf_args = partial(\n",
    "    TRAINER_ARGS,\n",
    "    output_dir=f\"./checkpoints/{os.getenv('WANDB_RUN_GROUP')}/{bidaf_run_name}\",\n",
    "    num_train_epochs=18,\n",
    "    per_device_train_batch_size=60,\n",
    "    per_device_eval_batch_size=60,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training and validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the baseline model, we are going to perform both training and validation (to observe the evolution of metrics over unseen data)$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bidaf_model = model.QABiDAFModel(embedding_layer, device=DEVICE)\n",
    "print(f\"The BiDAF model has {bidaf_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T14:31:14.109815Z",
     "iopub.status.busy": "2021-01-24T14:31:14.109485Z",
     "iopub.status.idle": "2021-01-24T14:31:14.179729Z",
     "shell.execute_reply": "2021-01-24T14:31:14.178558Z",
     "shell.execute_reply.started": "2021-01-24T14:31:14.109772Z"
    }
   },
   "outputs": [],
   "source": [
    "bidaf_optimizer = optim.Adadelta(bidaf_model.parameters(), lr=0.5)\n",
    "bidaf_lr_scheduler = transformers.get_constant_schedule(bidaf_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T14:31:16.104244Z",
     "iopub.status.busy": "2021-01-24T14:31:16.103907Z",
     "iopub.status.idle": "2021-01-24T14:31:16.187474Z",
     "shell.execute_reply": "2021-01-24T14:31:16.186315Z",
     "shell.execute_reply.started": "2021-01-24T14:31:16.104201Z"
    }
   },
   "outputs": [],
   "source": [
    "bidaf_trainer = training.SquadTrainer(\n",
    "    model=bidaf_model,\n",
    "    args=bidaf_args(run_name=bidaf_run_name),\n",
    "    data_collator=recurrent_dm.tokenizer,\n",
    "    train_dataset=recurrent_dm.train_dataset,\n",
    "    eval_dataset=recurrent_dm.val_dataset,\n",
    "    optimizers=(bidaf_optimizer, bidaf_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bidaf_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the baseline model, we will use the entire dataset to perform training (no validation)$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bidaf_model = model.QABiDAFModel(embedding_layer, device=DEVICE)\n",
    "print(f\"The BiDAF model has {bidaf_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bidaf_optimizer = optim.Adadelta(bidaf_model.parameters(), lr=0.5)\n",
    "bidaf_lr_scheduler = transformers.get_constant_schedule(bidaf_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bidaf_trainer = training.SquadTrainer(\n",
    "    model=bidaf_model,\n",
    "    args=bidaf_args(run_name=f\"{bidaf_run_name}-whole\", evaluation_strategy=\"no\"),\n",
    "    data_collator=recurrent_dm.tokenizer,\n",
    "    train_dataset=recurrent_dm.whole_dataset,\n",
    "    optimizers=(bidaf_optimizer, bidaf_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bidaf_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-26T14:05:03.100832Z",
     "iopub.status.busy": "2021-01-26T14:05:03.100385Z",
     "iopub.status.idle": "2021-01-26T14:05:03.168305Z",
     "shell.execute_reply": "2021-01-26T14:05:03.166774Z",
     "shell.execute_reply.started": "2021-01-26T14:05:03.100786Z"
    }
   },
   "source": [
    "As with the baseline model, we will predict answers for each question in the given test dataset and save them on a `JSON` file$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bidaf_test_output = bidaf_trainer.predict(recurrent_dm.test_dataset)\n",
    "bidaf_test_output.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T18:02:35.601859Z",
     "iopub.status.busy": "2021-01-21T18:02:35.601360Z",
     "iopub.status.idle": "2021-01-21T18:02:44.909008Z",
     "shell.execute_reply": "2021-01-21T18:02:44.907592Z",
     "shell.execute_reply.started": "2021-01-21T18:02:35.601819Z"
    }
   },
   "outputs": [],
   "source": [
    "bidaf_answers_path = \"results/answers/bidaf.json\"\n",
    "utils.save_answers(bidaf_answers_path, bidaf_test_output.predictions[-1])\n",
    "wandb.save(bidaf_answers_path);\n",
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section contains blocks of code that are used to train question answering models which are based on Transformer networks (`BERT`-related in our case). The models that we exploited are the following:\n",
    "- BERT (Bidirectional Encoder Representations from Transformers)\n",
    "- DistilBERT (a distilled version of BERT)\n",
    "- ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately)\n",
    "\n",
    "Since pre-training such models is very expensive, we relied on pre-trained versions of them (where the pre-training tasks are different than question answering), publicly available through the [HuggingFace](https://huggingface.co/) [Transformers](https://huggingface.co/transformers/) library. Pre-trained models are wrapped into an ad-hoc `PyTorch` module, which attaches to them the output layer to be used for question answering.\n",
    "\n",
    "Check the corresponding section to have a high-level view of each model$\\dots$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tokenizer that we are using for Transformer-based modules splits words using the `WordPiece` algorithm, removes accents and applies a lowercasing function to all the tokens, while also merging together questions and contexts as $[CLS] q_1 q_2 \\dots q_n [SEP] c_1 c_2 \\dots c_m [SEP]$ (it leverages the special tokens `[CLS]` and `[SEP]`). Moreover, the combined question/context sentence is truncated to a maximum number of tokens ($512$) and padded to the right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:48.996797Z",
     "iopub.status.busy": "2021-01-24T17:19:48.996482Z",
     "iopub.status.idle": "2021-01-24T17:19:49.082133Z",
     "shell.execute_reply": "2021-01-24T17:19:49.081032Z",
     "shell.execute_reply.started": "2021-01-24T17:19:48.996759Z"
    }
   },
   "outputs": [],
   "source": [
    "transformer_tokenizer = tokenizer.get_transformer_tokenizer(\n",
    "    config.BERT_VOCAB_PATH, config.MAX_BERT_TOKENS, device=DEVICE\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the recurrent-related part of the notebook, the `SquadDataManager` class pre-processes inputs by throwing away \"dirty\" and \"lost\" answers and then groups answers related to the same question and context pair into a single row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:49.219590Z",
     "iopub.status.busy": "2021-01-24T17:19:49.219262Z",
     "iopub.status.idle": "2021-01-24T17:19:49.333487Z",
     "shell.execute_reply": "2021-01-24T17:19:49.332291Z",
     "shell.execute_reply.started": "2021-01-24T17:19:49.219552Z"
    }
   },
   "outputs": [],
   "source": [
    "transformer_dm = dataset.SquadDataManager(\n",
    "    squad_dataset, transformer_tokenizer, val_split=config.VAL_SPLIT, device=DEVICE\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the training split of the whole dataset$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:50.393325Z",
     "iopub.status.busy": "2021-01-24T17:19:50.392999Z",
     "iopub.status.idle": "2021-01-24T17:19:50.470597Z",
     "shell.execute_reply": "2021-01-24T17:19:50.469490Z",
     "shell.execute_reply.started": "2021-01-24T17:19:50.393287Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>question</th>\n",
       "      <th>title</th>\n",
       "      <th>context_id</th>\n",
       "      <th>context</th>\n",
       "      <th>answer</th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56de4d9ecffd8e1900b4b7e2</td>\n",
       "      <td>What year was the Banská Akadémia founded?</td>\n",
       "      <td>Institute_of_technology</td>\n",
       "      <td>1860</td>\n",
       "      <td>The world's first institution of technology or...</td>\n",
       "      <td>[1735]</td>\n",
       "      <td>[167]</td>\n",
       "      <td>[171]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>572781a5f1498d1400e8fa1f</td>\n",
       "      <td>Who is elected every even numbered year?</td>\n",
       "      <td>Ann_Arbor,_Michigan</td>\n",
       "      <td>10585</td>\n",
       "      <td>Ann Arbor has a council-manager form of govern...</td>\n",
       "      <td>[mayor]</td>\n",
       "      <td>[192]</td>\n",
       "      <td>[197]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5730bb058ab72b1400f9c72c</td>\n",
       "      <td>Where were the use of advanced materials and t...</td>\n",
       "      <td>Sumer</td>\n",
       "      <td>17505</td>\n",
       "      <td>The most impressive and famous of Sumerian bui...</td>\n",
       "      <td>[Sumerian temples and palaces]</td>\n",
       "      <td>[421]</td>\n",
       "      <td>[449]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                question_id  \\\n",
       "0  56de4d9ecffd8e1900b4b7e2   \n",
       "1  572781a5f1498d1400e8fa1f   \n",
       "2  5730bb058ab72b1400f9c72c   \n",
       "\n",
       "                                            question                    title  \\\n",
       "0         What year was the Banská Akadémia founded?  Institute_of_technology   \n",
       "1           Who is elected every even numbered year?      Ann_Arbor,_Michigan   \n",
       "2  Where were the use of advanced materials and t...                    Sumer   \n",
       "\n",
       "   context_id                                            context  \\\n",
       "0        1860  The world's first institution of technology or...   \n",
       "1       10585  Ann Arbor has a council-manager form of govern...   \n",
       "2       17505  The most impressive and famous of Sumerian bui...   \n",
       "\n",
       "                           answer answer_start answer_end  \n",
       "0                          [1735]        [167]      [171]  \n",
       "1                         [mayor]        [192]      [197]  \n",
       "2  [Sumerian temples and palaces]        [421]      [449]  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformer_dm.train_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the validation split of the whole dataset$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:50.684113Z",
     "iopub.status.busy": "2021-01-24T17:19:50.683802Z",
     "iopub.status.idle": "2021-01-24T17:19:50.754943Z",
     "shell.execute_reply": "2021-01-24T17:19:50.753846Z",
     "shell.execute_reply.started": "2021-01-24T17:19:50.684075Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question_id</th>\n",
       "      <th>question</th>\n",
       "      <th>title</th>\n",
       "      <th>context_id</th>\n",
       "      <th>context</th>\n",
       "      <th>answer</th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>572674a05951b619008f7319</td>\n",
       "      <td>What is another speed that can also be reporte...</td>\n",
       "      <td>Film_speed</td>\n",
       "      <td>9354</td>\n",
       "      <td>The standard specifies how speed ratings shoul...</td>\n",
       "      <td>[SOS-based speed]</td>\n",
       "      <td>[793]</td>\n",
       "      <td>[808]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                question_id  \\\n",
       "0  572674a05951b619008f7319   \n",
       "\n",
       "                                            question       title  context_id  \\\n",
       "0  What is another speed that can also be reporte...  Film_speed        9354   \n",
       "\n",
       "                                             context             answer  \\\n",
       "0  The standard specifies how speed ratings shoul...  [SOS-based speed]   \n",
       "\n",
       "  answer_start answer_end  \n",
       "0        [793]      [808]  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformer_dm.val_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the same for the test dataset$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:19:51.479187Z",
     "iopub.status.busy": "2021-01-24T17:19:51.478866Z",
     "iopub.status.idle": "2021-01-24T17:19:51.545725Z",
     "shell.execute_reply": "2021-01-24T17:19:51.544640Z",
     "shell.execute_reply.started": "2021-01-24T17:19:51.479149Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>answer</th>\n",
       "      <th>answer_start</th>\n",
       "      <th>answer_end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [index, answer, answer_start, answer_end]\n",
       "Index: []"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformer_dm.test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The BERT model is a bidirectional transformer pretrained using a combination of masked language modeling objective and next sentence prediction on a large corpus comprising the Toronto Book Corpus and Wikipedia.\n",
    "\n",
    "The abstract from the paper is the following:\n",
    "\n",
    "> We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-26T13:59:00.352983Z",
     "iopub.status.busy": "2021-01-26T13:59:00.352544Z",
     "iopub.status.idle": "2021-01-26T13:59:00.419422Z",
     "shell.execute_reply": "2021-01-26T13:59:00.418165Z",
     "shell.execute_reply.started": "2021-01-26T13:59:00.352936Z"
    }
   },
   "source": [
    "Hyperparameters are taken directly from the BERT paper (in the section related to fine-tuning for the `SQuAD` dataset):\n",
    "- Epochs: $3$\n",
    "- Batch size: $32$ (we went for smaller sizes because of resources limitations)\n",
    "- Learning rate: $5e-5$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:28:00.099393Z",
     "iopub.status.busy": "2021-01-24T17:28:00.098954Z",
     "iopub.status.idle": "2021-01-24T17:28:00.175819Z",
     "shell.execute_reply": "2021-01-24T17:28:00.174666Z",
     "shell.execute_reply.started": "2021-01-24T17:28:00.099349Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_RUN_GROUP=bert\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_RUN_GROUP=bert\n",
    "bert_run_name = utils.get_run_name()\n",
    "bert_args = partial(\n",
    "    TRAINER_ARGS,\n",
    "    output_dir=f\"./checkpoints/{os.getenv('WANDB_RUN_GROUP')}/{bert_run_name}\",\n",
    "    num_train_epochs=3,\n",
    "    per_device_train_batch_size=16,\n",
    "    per_device_eval_batch_size=16,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training and validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with recurrent-based models, we are going to perform training and one validation run after each epoch, to observe metrics$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-30T16:00:52.310767Z",
     "iopub.status.busy": "2021-01-30T16:00:52.310302Z",
     "iopub.status.idle": "2021-01-30T16:01:01.872987Z",
     "shell.execute_reply": "2021-01-30T16:01:01.871623Z",
     "shell.execute_reply.started": "2021-01-30T16:00:52.310720Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The BERT model has 114208512 parameters\n"
     ]
    }
   ],
   "source": [
    "bert_model = model.QABertModel(device=DEVICE)\n",
    "print(f\"The BERT model has {bert_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:32:16.101977Z",
     "iopub.status.busy": "2021-01-24T17:32:16.101668Z",
     "iopub.status.idle": "2021-01-24T17:32:16.169743Z",
     "shell.execute_reply": "2021-01-24T17:32:16.168517Z",
     "shell.execute_reply.started": "2021-01-24T17:32:16.101939Z"
    }
   },
   "outputs": [],
   "source": [
    "bert_optimizer = optim.Adam(bert_model.parameters(), lr=5e-5)\n",
    "bert_lr_scheduler = transformers.get_constant_schedule(bert_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-24T17:32:16.172085Z",
     "iopub.status.busy": "2021-01-24T17:32:16.171773Z",
     "iopub.status.idle": "2021-01-24T17:32:16.256158Z",
     "shell.execute_reply": "2021-01-24T17:32:16.254702Z",
     "shell.execute_reply.started": "2021-01-24T17:32:16.172047Z"
    }
   },
   "outputs": [],
   "source": [
    "bert_trainer = training.SquadTrainer(\n",
    "    model=bert_model,\n",
    "    args=bert_args(run_name=bert_run_name),\n",
    "    data_collator=transformer_dm.tokenizer,\n",
    "    train_dataset=transformer_dm.train_dataset,\n",
    "    eval_dataset=transformer_dm.val_dataset,\n",
    "    optimizers=(bert_optimizer, bert_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with recurrent-based modules, we are going to perform training over the whole dataset (with no validation at all)$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_model = model.QABertModel(device=DEVICE)\n",
    "print(f\"The BERT model has {bert_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_optimizer = optim.Adam(bert_model.parameters(), lr=5e-5)\n",
    "bert_lr_scheduler = transformers.get_constant_schedule(bert_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_trainer = training.SquadTrainer(\n",
    "    model=bert_model,\n",
    "    args=bert_args(run_name=f\"{bert_run_name}-whole\", evaluation_strategy=\"no\"),\n",
    "    data_collator=transformer_dm.tokenizer,\n",
    "    train_dataset=transformer_dm.whole_dataset,\n",
    "    optimizers=(bert_optimizer, bert_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with recurrent-based modules, we will predict one answer for each question in the test dataset and save results in a `JSON` file$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_test_output = bert_trainer.predict(transformer_dm.test_dataset)\n",
    "bert_test_output.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T17:57:02.820786Z",
     "iopub.status.busy": "2021-01-21T17:57:02.820061Z",
     "iopub.status.idle": "2021-01-21T17:58:58.223064Z",
     "shell.execute_reply": "2021-01-21T17:58:58.221615Z",
     "shell.execute_reply.started": "2021-01-21T17:57:02.820720Z"
    }
   },
   "outputs": [],
   "source": [
    "bert_answers_path = \"results/answers/bert.json\"\n",
    "utils.save_answers(bert_answers_path, bert_test_output.predictions[-1])\n",
    "wandb.save(bert_answers_path);\n",
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DistilBERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than _bert-base-uncased_, runs 60% faster while preserving over 95% of BERT’s performances as measured on the GLUE language understanding benchmark.\n",
    "\n",
    "The abstract from the paper is the following:\n",
    "\n",
    "> As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pretraining phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pretraining, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperparameters are the same as the ones used for fine-tuning BERT$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T17:56:05.140129Z",
     "iopub.status.busy": "2021-01-21T17:56:05.139775Z",
     "iopub.status.idle": "2021-01-21T17:56:05.227681Z",
     "shell.execute_reply": "2021-01-21T17:56:05.226548Z",
     "shell.execute_reply.started": "2021-01-21T17:56:05.140090Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_RUN_GROUP=bert\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_RUN_GROUP=distilbert\n",
    "distilbert_run_name = utils.get_run_name()\n",
    "distilbert_args = partial(\n",
    "    TRAINER_ARGS,\n",
    "    output_dir=f\"./checkpoints/{os.getenv('WANDB_RUN_GROUP')}/{distilbert_run_name}\",\n",
    "    num_train_epochs=3,\n",
    "    per_device_train_batch_size=16,\n",
    "    per_device_eval_batch_size=16,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training and validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same reasoning as above$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T17:55:53.705471Z",
     "iopub.status.busy": "2021-01-21T17:55:53.705021Z",
     "iopub.status.idle": "2021-01-21T17:56:03.103337Z",
     "shell.execute_reply": "2021-01-21T17:56:03.101497Z",
     "shell.execute_reply.started": "2021-01-21T17:55:53.705426Z"
    }
   },
   "outputs": [],
   "source": [
    "distilbert_model = model.QADistilBertModel(device=DEVICE)\n",
    "print(f\"The DistilBERT model has {distilbert_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_optimizer = optim.Adam(distilbert_model.parameters(), lr=5e-5)\n",
    "distilbert_lr_scheduler = transformers.get_constant_schedule(distilbert_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_trainer = training.SquadTrainer(\n",
    "    model=distilbert_model,\n",
    "    args=distilbert_args(run_name=distilbert_run_name),\n",
    "    data_collator=transformer_dm.tokenizer,\n",
    "    train_dataset=transformer_dm.train_dataset,\n",
    "    eval_dataset=transformer_dm.val_dataset,\n",
    "    optimizers=(distilbert_optimizer, distilbert_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same reasoning as above$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_model = model.QADistilBertModel(device=DEVICE)\n",
    "print(f\"The DistilBERT model has {distilbert_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_optimizer = optim.Adam(distilbert_model.parameters(), lr=5e-5)\n",
    "distilbert_lr_scheduler = transformers.get_constant_schedule(distilbert_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_trainer = training.SquadTrainer(\n",
    "    model=distilbert_model,\n",
    "    args=distilbert_args(run_name=f\"{distilbert_run_name}-whole\", evaluation_strategy=\"no\"),\n",
    "    data_collator=transformer_dm.tokenizer,\n",
    "    train_dataset=transformer_dm.whole_dataset,\n",
    "    optimizers=(distilbert_optimizer, distilbert_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same reasoning as above$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distilbert_test_output = distilbert_trainer.predict(transformer_dm.test_dataset)\n",
    "distilbert_test_output.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T17:57:02.820786Z",
     "iopub.status.busy": "2021-01-21T17:57:02.820061Z",
     "iopub.status.idle": "2021-01-21T17:58:58.223064Z",
     "shell.execute_reply": "2021-01-21T17:58:58.221615Z",
     "shell.execute_reply.started": "2021-01-21T17:57:02.820720Z"
    }
   },
   "outputs": [],
   "source": [
    "distilbert_answers_path = \"results/answers/distilbert.json\"\n",
    "utils.save_answers(distilbert_answers_path, distilbert_test_output.predictions[-1])\n",
    "wandb.save(distilbert_answers_path);\n",
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELECTRA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ELECTRA is a new pretraining approach which trains two transformer models: the generator and the discriminator. The generator’s role is to replace tokens in a sequence, and is therefore trained as a masked language model. The discriminator, which is the model we’re interested in, tries to identify which tokens were replaced by the generator in the sequence.\n",
    "\n",
    "The abstract from the paper is the following:\n",
    "\n",
    "> Masked language modeling (MLM) pretraining methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pretraining task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pretraining task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-26T13:59:49.652513Z",
     "iopub.status.busy": "2021-01-26T13:59:49.652070Z",
     "iopub.status.idle": "2021-01-26T13:59:49.719222Z",
     "shell.execute_reply": "2021-01-26T13:59:49.717705Z",
     "shell.execute_reply.started": "2021-01-26T13:59:49.652466Z"
    }
   },
   "source": [
    "Hyperparameters are the same as the ones used for fine-tuning BERT and DistilBERT$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T17:56:05.140129Z",
     "iopub.status.busy": "2021-01-21T17:56:05.139775Z",
     "iopub.status.idle": "2021-01-21T17:56:05.227681Z",
     "shell.execute_reply": "2021-01-21T17:56:05.226548Z",
     "shell.execute_reply.started": "2021-01-21T17:56:05.140090Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: WANDB_RUN_GROUP=bert\n"
     ]
    }
   ],
   "source": [
    "%env WANDB_RUN_GROUP=electra\n",
    "electra_run_name = utils.get_run_name()\n",
    "electra_args = partial(\n",
    "    TRAINER_ARGS,\n",
    "    output_dir=f\"./checkpoints/{os.getenv('WANDB_RUN_GROUP')}/{electra_run_name}\",\n",
    "    num_train_epochs=3,\n",
    "    per_device_train_batch_size=16,\n",
    "    per_device_eval_batch_size=16,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training and validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same reasoning as above$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T17:55:53.705471Z",
     "iopub.status.busy": "2021-01-21T17:55:53.705021Z",
     "iopub.status.idle": "2021-01-21T17:56:03.103337Z",
     "shell.execute_reply": "2021-01-21T17:56:03.101497Z",
     "shell.execute_reply.started": "2021-01-21T17:55:53.705426Z"
    }
   },
   "outputs": [],
   "source": [
    "electra_model = model.QAElectraModel(device=DEVICE)\n",
    "print(f\"The ELECTRA model has {electra_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_optimizer = optim.Adam(electra_model.parameters(), lr=5e-5)\n",
    "electra_lr_scheduler = transformers.get_constant_schedule(electra_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_trainer = training.SquadTrainer(\n",
    "    model=electra_model,\n",
    "    args=electra_args(run_name=electra_run_name),\n",
    "    data_collator=transformer_dm.tokenizer,\n",
    "    train_dataset=transformer_dm.train_dataset,\n",
    "    eval_dataset=transformer_dm.val_dataset,\n",
    "    optimizers=(electra_optimizer, electra_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same reasoning as above$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_model = model.QAElectraModel(device=DEVICE)\n",
    "print(f\"The ELECTRA model has {electra_model.count_parameters()} parameters\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_optimizer = optim.Adam(electra_model.parameters(), lr=5e-5)\n",
    "electra_lr_scheduler = transformers.get_constant_schedule(electra_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_trainer = training.SquadTrainer(\n",
    "    model=electra_model,\n",
    "    args=electra_args(run_name=f\"{electra_run_name}-whole\", evaluation_strategy=\"no\"),\n",
    "    data_collator=transformer_dm.tokenizer,\n",
    "    train_dataset=transformer_dm.whole_dataset,\n",
    "    optimizers=(electra_optimizer, electra_lr_scheduler),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same reasoning as above$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "electra_test_output = electra_trainer.predict(transformer_dm.test_dataset)\n",
    "electra_test_output.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-21T17:57:02.820786Z",
     "iopub.status.busy": "2021-01-21T17:57:02.820061Z",
     "iopub.status.idle": "2021-01-21T17:58:58.223064Z",
     "shell.execute_reply": "2021-01-21T17:58:58.221615Z",
     "shell.execute_reply.started": "2021-01-21T17:57:02.820720Z"
    }
   },
   "outputs": [],
   "source": [
    "electra_answers_path = \"results/answers/electra.json\"\n",
    "utils.save_answers(electra_answers_path, electra_test_output.predictions[-1])\n",
    "wandb.save(electra_answers_path);\n",
    "wandb.finish()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}