SBERT_CORD19_QA_Roberta.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T28QEDT-Uqj2"
   },
   "source": [
    "<p align=\"center\">\n",
    " <img src=\"http://www.di.uoa.gr/themes/corporate_lite/logo_el.png\" title=\"Department of Informatics and Telecommunications - University of Athens\"/> </p>\n",
    "\n",
    "---\n",
    "<h1 align=\"center\"> \n",
    "  Artificial Intelligence\n",
    "</h1>\n",
    "<h1 align=\"center\" > \n",
    "  Deep Learning for Natural Language Processing\n",
    "</h1>\n",
    "\n",
    "---\n",
    "<h2 align=\"center\"> \n",
    " <b>Konstantinos Nikoletos</b>\n",
    "</h2>\n",
    "\n",
    "<h3 align=\"center\"> \n",
    " <b>Winter 2020-2021</b>\n",
    "</h3>\n",
    "\n",
    "\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wk9SOEk6mCXZ"
   },
   "source": [
    "\n",
    "### __Task__ \n",
    "This exercise is about developing a document retrieval system to return titles of scientific\n",
    "papers containing the answer to a given user question. You will use the first version of\n",
    "the COVID-19 Open Research Dataset (CORD-19) in your work (articles in the folder\n",
    "comm use subset).\n",
    "\n",
    "\n",
    "For example, for the question “What are the coronaviruses?”, your system can return the\n",
    "paper title “Distinct Roles for Sialoside and Protein Receptors in Coronavirus Infection”\n",
    "since this paper contains the answer to the asked question.\n",
    "\n",
    "\n",
    "To achieve the goal of this exercise, you will need first to read the paper Sentence-BERT:\n",
    "Sentence Embeddings using Siamese BERT-Networks, in order to understand how you\n",
    "can create sentence embeddings. In the related work of this paper, you will also find other\n",
    "approaches for developing your model. For example, you can using Glove embeddings,\n",
    "etc. In this link, you can find the extended versions of this dataset to test your model, if\n",
    "you want. You are required to:\n",
    "\n",
    "\n",
    "<ol type=\"a\">\n",
    "  <li>Preprocess the provided dataset. You will decide which data of each paper is useful\n",
    "to your model in order to create the appropriate embeddings. You need to explain\n",
    "your decisions.</li>\n",
    "  <li>Implement at least 2 different sentence embedding approaches (see the related work\n",
    "of the Sentence-BERT paper), in order for your model to retrieve the titles of the\n",
    "papers related to a given question.</li>\n",
    "  <li>Compare your 2 models based on at least 2 different criteria of your choice. Explain\n",
    "why you selected these criteria, your implementation choices, and the results. Some\n",
    "questions you can pose are included here. You will need to provide the extra questions\n",
    "you posed to your model and the results of all the questions as well.</li>\n",
    "</ol>\n",
    "\n",
    "### __Notebook__ \n",
    "\n",
    "\n",
    "In this notebook I am going to implement Sentence Bert for the CORD-19 dataset\n",
    "\n",
    "\n",
    "---\n",
    "---\n",
    "\n",
    "***Implemented in colab (dark mode)***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6OfAccyufasd"
   },
   "source": [
    "__Import__ of essential libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dVSHtKTrfGiV",
    "outputId": "fad907b8-fba4-4cd1-f7fd-6e8c0f0715ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /root/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd \n",
    "import sys # only needed to determine Python version number\n",
    "import matplotlib # only needed to determine Matplotlib version \n",
    "import nltk\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import pprint\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torchtext import data\n",
    "import logging\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('averaged_perceptron_tagger')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vC9DWskzConr"
   },
   "source": [
    "Selecting device (GPU - CUDA if available)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NiRkpLhiCntl",
    "outputId": "2a14589c-f7b7-49be-f0d0-4013d3cdea4d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on GPU.\n"
     ]
    }
   ],
   "source": [
    "# First checking if GPU is available\n",
    "train_on_gpu=torch.cuda.is_available()\n",
    "\n",
    "if(train_on_gpu):\n",
    "    print('Training on GPU.')\n",
    "else:\n",
    "    print('No GPU available, training on CPU.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cvK_TWa2YNg5"
   },
   "source": [
    "# Loading data\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "D6nnb0BFMZDx",
    "outputId": "be1a7b40-1c34-4d5a-fed5-e910c514df27"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "# Opening data file\n",
    "import io\n",
    "from google.colab import drive\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import json\n",
    "\n",
    "drive.mount('/content/drive',force_remount=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wih8b1VKVeS2"
   },
   "source": [
    "Loading the dictionary if it has been created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "a8m_oar6ZzH3"
   },
   "outputs": [],
   "source": [
    "#@title Select number of papers that will be feeded in the model { vertical-output: true, display-mode: \"both\" }\n",
    "number_of_papers = \"9000\" #@param [\"1000\", \"3000\", \"6000\", \"9000\"]\n",
    "import pickle\n",
    "\n",
    "CORD19_Dataframe = r\"/content/drive/My Drive/AI_4/CORD19_SentenceMap_\"+number_of_papers+\".pkl\"\n",
    "with open(CORD19_Dataframe, 'rb') as drivef:\n",
    "  CORD19Dictionary = pickle.load(drivef)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2xLN9x3B6KfR"
   },
   "source": [
    "OR the summary of the papers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A2Qk-Pnn6Kfg"
   },
   "outputs": [],
   "source": [
    "#@title Select number of summarized papers that will be feeded in the model { vertical-output: true, display-mode: \"both\" }\n",
    "number_of_papers = \"9000\" #@param [\"1000\", \"3000\", \"6000\", \"9000\"]\n",
    "import pickle\n",
    "\n",
    "CORD19_Dataframe = r\"/content/drive/My Drive/AI_4/CORD19_SentenceMap_Summarized_\"+number_of_papers+\".pkl\"\n",
    "with open(CORD19_Dataframe, 'rb') as drivef:\n",
    "  CORD19Dictionary = pickle.load(drivef)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aJ16If_pdTXV"
   },
   "source": [
    "## Queries\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PdGaXWMdWIdh"
   },
   "outputs": [],
   "source": [
    "query_list = [\n",
    "  'What are the coronoviruses?',\n",
    "  'What was discovered in Wuhuan in December 2019?',\n",
    "  'What is Coronovirus Disease 2019?',\n",
    "  'What is COVID-19?',\n",
    "  'What is caused by SARS-COV2?', 'How is COVID-19 spread?',\n",
    "  'Where was COVID-19 discovered?','How does coronavirus spread?'\n",
    "]\n",
    "\n",
    "proposed_answers = [\n",
    "  'Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes. ',\n",
    "  'In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.',\n",
    "  'Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.',\n",
    "  'COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.',\n",
    "  'Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.', \n",
    "  'First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.',\n",
    "  'In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.',\n",
    "  'The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.'\n",
    "]\n",
    "\n",
    "myquery_list = [\n",
    "  \"How long can the coronavirus survive on surfaces?\",\n",
    "  \"What means COVID-19?\",\n",
    "  \"Is COVID19 worse than flue?\",\n",
    "  \"When the vaccine will be ready?\",\n",
    "  \"Whats the proteins that consist COVID-19?\",\n",
    "  \"Whats the symptoms of COVID-19?\",\n",
    "  \"How can I prevent COVID-19?\",\n",
    "  \"What treatments are available for COVID-19?\",\n",
    "  \"Is hand sanitizer effective against COVID-19?\",\n",
    "  \"Am I at risk for serious complications from COVID-19 if I smoke cigarettes?\",\n",
    "  \"Are there any FDA-approved drugs (medicines) for COVID-19?\",\n",
    "  \"How are people tested?\",\n",
    "  \"Why is the disease being called coronavirus disease 2019, COVID-19?\",\n",
    "  \"Am I at risk for COVID-19 from mail, packages, or products?\",\n",
    "  \"What is community spread?\",\n",
    "  \"How can I protect myself?\",\n",
    "  \"What is a novel coronavirus?\",\n",
    "  \"Was Harry Potter a good magician?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nwPB-YNeW6t2"
   },
   "source": [
    "# Results dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "_kBsV1FlW_4y",
    "outputId": "d444e1ff-6799-443e-ace1-1bbf44417c93"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and anim...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emer...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SA...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airb...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...               NaN\n",
       "1  What was discovered in Wuhuan in December 2019?  ...               NaN\n",
       "2                What is Coronovirus Disease 2019?  ...               NaN\n",
       "3                                What is COVID-19?  ...               NaN\n",
       "4                     What is caused by SARS-COV2?  ...               NaN\n",
       "5                          How is COVID-19 spread?  ...               NaN\n",
       "6                   Where was COVID-19 discovered?  ...               NaN\n",
       "7                     How does coronavirus spread?  ...               NaN\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resultsDf = pd.DataFrame(columns=['Number of papers','Embeddings creation time'])\n",
    "\n",
    "queriesDf = pd.DataFrame(columns=['Query','Proposed_answer','Model_answer','Cosine_similarity'])\n",
    "queriesDf['Query'] = query_list\n",
    "queriesDf['Proposed_answer'] = proposed_answers\n",
    "\n",
    "myQueriesDf = pd.DataFrame(columns=['Query','Model_answer','Cosine_similarity'])\n",
    "myQueriesDf['Query'] = myquery_list\n",
    "\n",
    "queriesDf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ODeyHCYINWRR"
   },
   "source": [
    "# SBERT\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "J7g1wtxuwlkx",
    "outputId": "f7b3a7ba-cb61-417d-888a-bf0313e12ba2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: sentence-transformers in /usr/local/lib/python3.6/dist-packages (0.4.1.2)\n",
      "Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.4.1)\n",
      "Requirement already satisfied, skipping upgrade: transformers<5.0.0,>=3.1.0 in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (4.3.2)\n",
      "Requirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (4.41.1)\n",
      "Requirement already satisfied, skipping upgrade: nltk in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (3.2.5)\n",
      "Requirement already satisfied, skipping upgrade: torch>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.7.0+cu101)\n",
      "Requirement already satisfied, skipping upgrade: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (0.22.2.post1)\n",
      "Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.19.5)\n",
      "Requirement already satisfied, skipping upgrade: sentencepiece in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (0.1.95)\n",
      "Requirement already satisfied, skipping upgrade: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (0.0.43)\n",
      "Requirement already satisfied, skipping upgrade: filelock in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (3.0.12)\n",
      "Requirement already satisfied, skipping upgrade: packaging in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (20.9)\n",
      "Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (2.23.0)\n",
      "Requirement already satisfied, skipping upgrade: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (2019.12.20)\n",
      "Requirement already satisfied, skipping upgrade: tokenizers<0.11,>=0.10.1 in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (0.10.1)\n",
      "Requirement already satisfied, skipping upgrade: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (0.8)\n",
      "Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (3.4.0)\n",
      "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from nltk->sentence-transformers) (1.15.0)\n",
      "Requirement already satisfied, skipping upgrade: future in /usr/local/lib/python3.6/dist-packages (from torch>=1.6.0->sentence-transformers) (0.16.0)\n",
      "Requirement already satisfied, skipping upgrade: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch>=1.6.0->sentence-transformers) (3.7.4.3)\n",
      "Requirement already satisfied, skipping upgrade: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sentence-transformers) (1.0.0)\n",
      "Requirement already satisfied, skipping upgrade: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers<5.0.0,>=3.1.0->sentence-transformers) (7.1.2)\n",
      "Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers<5.0.0,>=3.1.0->sentence-transformers) (2.4.7)\n",
      "Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (2.10)\n",
      "Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (3.0.4)\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (2020.12.5)\n",
      "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (1.24.3)\n",
      "Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers<5.0.0,>=3.1.0->sentence-transformers) (3.4.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install -U sentence-transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_ba6VdQFfsWn"
   },
   "source": [
    "# Sentence Embeddings using Siamese BERT-Networks\n",
    "---\n",
    "__How does SentenceBERT work?__\n",
    "\n",
    "Let’s start by looking at the architecture of SentenceBERT, which I will call SBERT from here on. SBERT is a so-called twin network which allows it to process two sentences in the same way, simultaneously. These two twins are identical down to every parameter (their weight is tied), which allows us to think about this architecture as a single model used multiple times. (I think the reason for the twin formulation is due to mathematical ease, but I need to read the original paper for figuring it out)\n",
    "\n",
    "It becomes apparent from the image that BERT makes up the base of this model, to which a pooling layer has been appended. This pooling layer enables us to create a fixed-size representation for input sentences of varying lengths. The authors experimented with different pooling strategies; MEAN- and MAX pooling or utilising the CLS token BERT per default already generates. How these perform and compare will be discussed later.\n",
    "Since the purpose of creating these fixed-size sentence embeddings was to encode their semantics did the authors fine-tune their network on Semantic Textual Similarity data. \n",
    "\n",
    "<p align=\"center\">\n",
    " <img src=\"https://miro.medium.com/max/700/1*LGlrGO9b5Mt1X3-ZUyalng.png\" title=\"Department of Informatics and Telecommunications - University of Athens\"/> </p>\n",
    "\n",
    "\n",
    "Generally,\n",
    "\n",
    "__Sentence embeddings__ are a similar concept. It embeds a full sentence into a vector space. These sentence embeddings retain some nice properties, as they inherit features from their underlying word embeddings.\n",
    "There is a huge trend lately for a quest for Universal Embeddings: embeddings that are pre-trained on a large corpus and can be plugged in a variety of downstream task models (sentimental analysis, classification, translation…) to automatically improve their performance by incorporating some general word/sentence representations learned on the larger dataset.\n",
    "While unsupervised representation learning of sentences had been the norm for quite some time, last year has seen a shift toward supervised and multi-task learning schemes with a number of very interesting proposals in late 2017/early 2018.\n",
    "\n",
    "\n",
    "\n",
    " Creating the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zdc_ggRWNo5C"
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer, util\n",
    "import torch\n",
    "import time\n",
    "\n",
    "embedder = SentenceTransformer('paraphrase-distilroberta-base-v1', device='cuda')\n",
    "# embedder = SentenceTransformer('stsb-roberta-large', device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V5KbffTGfE9I"
   },
   "source": [
    "# Initializing corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l_L7TL9AfECp"
   },
   "outputs": [],
   "source": [
    "# Encoding the papers\n",
    "corpus = list(CORD19Dictionary.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FbDuU-ZVbp_z"
   },
   "source": [
    "# Creating the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 100,
     "referenced_widgets": [
      "46c9e683d3d74f95873119c1222986d9",
      "7f45b4f8e3aa47b0b8ecbccfd2f56c75",
      "5559cb96327544a3b40ae7a0aa5ff6f3",
      "b557158e8e284a76b5080c5155354bb7",
      "dc7eec0dc4734d358a5f0930b9227e6a",
      "627962abe74d4b519229906cb57e7ef3",
      "730c3e1d7d8649cc9ee495ad8a6e9996",
      "47bce05c5be7463fa80c63547f647939"
     ]
    },
    "id": "cq6N8lvjr6d8",
    "outputId": "7530df0c-0be0-4c95-e9bf-4db3b7063b6e"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46c9e683d3d74f95873119c1222986d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Batches', max=6793.0, style=ProgressStyle(description_wid…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "CPU times: user 1min 45s, sys: 35.7 s, total: 2min 21s\n",
      "Wall time: 2min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Encoding the papers\n",
    "corpus_embeddings = embedder.encode(corpus, convert_to_tensor=True,show_progress_bar=True, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ssq37-ZL6uU5"
   },
   "source": [
    "Saving corpus as tensors to drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "K1BJJofY6r8e"
   },
   "outputs": [],
   "source": [
    "corpus_embeddings_path = r\"/content/drive/My Drive/AI_4/corpus_embeddings_\"+number_of_papers+\".pt\"\n",
    "torch.save(corpus_embeddings,corpus_embeddings_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rv0gd5jB-rW2"
   },
   "source": [
    "# Loading embeddings if have been created and saved\n",
    "\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KnuBqIvkqyjp"
   },
   "outputs": [],
   "source": [
    "corpus_embeddings_path = r\"/content/drive/My Drive/AI_4/corpus_embeddings_\"+number_of_papers+\".pt\"\n",
    "with open(corpus_embeddings_path, 'rb') as f:\n",
    "    corpus_embeddings = torch.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OBnA24H--5AZ"
   },
   "source": [
    "# Evaluation\n",
    "---\n",
    "\n",
    "Printing the top 3 answers and storing to the dataframe the first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X0oozOLqlMv_"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from  nltk import tokenize\n",
    "from termcolor import colored\n",
    "\n",
    "def paperTitle(answer,SentenceMap):\n",
    "  record = SentenceMap[answer]\n",
    "  print(\"Paper title:\",record[1])\n",
    "  print(\"Paper id:   \",record[0])  \n",
    "\n",
    "# Find the closest sentence of the corpus for each query sentence based on cosine similarity\n",
    "def evaluation(topk,query_list,corpus_embeddings,resultsDf):\n",
    "  query_answers = []\n",
    "  scores = []\n",
    "  for query in query_list:\n",
    "      query_embedding = embedder.encode(query, convert_to_tensor=True, device='cuda')\n",
    "\n",
    "      # We use cosine-similarity and torch.topk to find the highest scores\n",
    "      cos_scores = util.pytorch_cos_sim(query_embedding, corpus_embeddings)[0]\n",
    "      top_results = torch.topk(cos_scores, k=top_k)\n",
    "\n",
    "      print(\"\\n\\n======================\\n\\n\")\n",
    "      print(\"Query:\",colored(query,'green') )\n",
    "\n",
    "      for iter, score, idx in zip(range(0,top_k),top_results[0], top_results[1]):\n",
    "        print(\"\\n-> \",iter+1)\n",
    "        answer = ' '.join([re.sub(r\"^\\[.*\\]\", \"\", x) for x in corpus[idx].split()])\n",
    "        if len(tokenize.word_tokenize(answer)) > 1:\n",
    "          print(\"Score: {:.4f}\".format(score))\n",
    "          paperTitle(corpus[idx],CORD19Dictionary)\n",
    "          print(\"Anser size: \",len(tokenize.word_tokenize(answer)))\n",
    "          print(\"Anser: \")\n",
    "          if iter==0:\n",
    "            scores.append(score.item())\n",
    "            query_answers.append(answer)\n",
    "          print(colored(answer,'yellow'))\n",
    "  resultsDf['Model_answer'] = query_answers\n",
    "  resultsDf['Cosine_similarity'] = scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F36KUgWNId-0",
    "outputId": "5472ec6d-e40c-4dbb-a3c8-a3a844d0b82c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat are the coronoviruses?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6327\n",
      "Paper title: Revisiting the dangers of the coronavirus in the ophthalmology practice\n",
      "Paper id:    804a9591280b7f64fa79cd3e4a9358976b084ffb\n",
      "Anser size:  6\n",
      "Anser: \n",
      "\u001b[33mCoronaviruses: what are they?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6319\n",
      "Paper title: The sialic acid binding activity of the S protein facilitates infection by porcine transmissible gastroenteritis coronavirus\n",
      "Paper id:    7902723eb8b21baa5eef8703832de11cc242a43b\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mThese coronaviruses can be differentiated in three distinct groups.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.6172\n",
      "Paper title: Avian viral surveillance in Victoria, Australia, and detection of two novel avian herpesviruses\n",
      "Paper id:    8780e9524d9271a7b8e789f0f8c4eb6860ca8c50\n",
      "Anser size:  3\n",
      "Anser: \n",
      "\u001b[33mAvian coronaviruses.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat was discovered in Wuhuan in December 2019?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.4792\n",
      "Paper title: Morbidity and Mortality Weekly Report\n",
      "Paper id:    11f13e2859eb22b349dbef68fd8124b8ba445aac\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mNine of the first 11 U.S. 2019-nCoV patients were exposed in Wuhan, China.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.4736\n",
      "Paper title: Transmission routes of 2019-nCoV and controls in dental practice\n",
      "Paper id:    9756bb3c608ed790d2306fc8db815a694eeca45f\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mAn emergent pneumonia outbreak originated in Wuhan City, in the late December 2019 1 .\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.4568\n",
      "Paper title: \n",
      "Paper id:    9267b60cb1ef8cf0cff3abb351e73dac5d7d7dea\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mInfection from an emerging pathogenic coronavirus was first reported in December 2019 in China.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is Coronovirus Disease 2019?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6755\n",
      "Paper title: Population genetics, community of parasites, and resistance to rodenticides in an urban brown rat (Rattus norvegicus) population\n",
      "Paper id:    c8d60caf44017989b3b9633350fc1d2efda570a5\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mCoronavirus.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6742\n",
      "Paper title: Cell Discovery Phase-adjusted estimation of the number of Coronavirus Disease 2019 cases in Wuhan, China\n",
      "Paper id:    6abb30ae61aa5e41f16a28b9437940d5d76d745b\n",
      "Anser size:  20\n",
      "Anser: \n",
      "\u001b[33mWorld Health Organization (WHO) now has named the disease Coronavirus Disease 2019 (COVID- 19) 3 .\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.6440\n",
      "Paper title: Tropical Medicine and Infectious Disease Potential Intermediate Hosts for Coronavirus Transmission: No Evidence of Clade 2c Coronaviruses in Domestic Livestock from Ghana\n",
      "Paper id:    95cc317541d97e3dbaa1662894fdbed842098910\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mcoronavirus.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6658\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6234\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5788\n",
      "Paper title: Clinical Medicine Optimization Method for Forecasting Confirmed Cases of COVID-19 in China\n",
      "Paper id:    1d7f8850c5244fdc9b387038e7eeae9bcbbde6d2\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mThe main dataset of this study is COVID-19 dataset.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is caused by SARS-COV2?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6612\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mIs there an effective specific anti-SARS-CoV-2 solution?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6456\n",
      "Paper title: Interdisciplinarity and Infectious Diseases: An Ebola Case Study\n",
      "Paper id:    38486671848611786384f2cc0a49066c5fca595f\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mWhat is the transmission cycle that caused the global SARS epidemic?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.6373\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mIs there any effect of SARS-CoV vaccine designed according to S protein on SARS-CoV-2?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow is COVID-19 spread?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6255\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mThe COVID-19 has then rapidly spread to all over China and the world.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6016\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  18\n",
      "Anser: \n",
      "\u001b[33mIt is found that the COVID-19 can be transmitted through droplets, contact, aerosol, etc.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5892\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhere was COVID-19 discovered?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5955\n",
      "Paper title: Clinical Medicine Optimization Method for Forecasting Confirmed Cases of COVID-19 in China\n",
      "Paper id:    1d7f8850c5244fdc9b387038e7eeae9bcbbde6d2\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mThe main dataset of this study is COVID-19 dataset.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5852\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5762\n",
      "Paper title: Comment\n",
      "Paper id:    861f05718b673306c2044167d92bb31d68a0d9bb\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mAll these figures can be seen emerging during the COVID-19 outbreak.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow does coronavirus spread?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6793\n",
      "Paper title: Transcriptional profiling of feline infectious peritonitis virus infection in CRFK cells and in PBMCs from FIP diagnosed cats\n",
      "Paper id:    d4b76917de146cdbe4c922c15ac3d87d3d483446\n",
      "Anser size:  9\n",
      "Anser: \n",
      "\u001b[33mThe pathogenesis of feline coronavirus infection is unclear.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6708\n",
      "Paper title: Analysis of the codon usage pattern in Middle East Respiratory Syndrome Coronavirus\n",
      "Paper id:    627ada5c21fb8d0e43b37999fa66bf41ca36c353\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mThis may hint that coronavirus does not spread so widely in humans.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.6524\n",
      "Paper title: Could human coronavirus OC43 have co-evolved with early humans?\n",
      "Paper id:    123db7e17bb17a0d4ff1e7ef87cb629d6ac78125\n",
      "Anser size:  3\n",
      "Anser: \n",
      "\u001b[33mHuman coronavirus evolution\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "  top_k = 3\n",
    "  evaluation(top_k,query_list,corpus_embeddings,queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UxYIV1MPKZBA",
    "outputId": "e09ae2cb-2666-4f00-935f-e4791c6e434d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow long can the coronavirus survive on surfaces?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6419\n",
      "Paper title: Transcriptional profiling of feline infectious peritonitis virus infection in CRFK cells and in PBMCs from FIP diagnosed cats\n",
      "Paper id:    d4b76917de146cdbe4c922c15ac3d87d3d483446\n",
      "Anser size:  9\n",
      "Anser: \n",
      "\u001b[33mThe pathogenesis of feline coronavirus infection is unclear.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6112\n",
      "Paper title: The Mongoose, the Pheasant, the Pox, and the Retrovirus\n",
      "Paper id:    06c172a11ee30931ac537dfb70ce020dced50918\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mHow then did this mammalian retrovirus get into birds?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5973\n",
      "Paper title: Analysis of the codon usage pattern in Middle East Respiratory Syndrome Coronavirus\n",
      "Paper id:    627ada5c21fb8d0e43b37999fa66bf41ca36c353\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mThis may hint that coronavirus does not spread so widely in humans.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat means COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6750\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6135\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5825\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  18\n",
      "Anser: \n",
      "\u001b[33mIt is found that the COVID-19 can be transmitted through droplets, contact, aerosol, etc.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mIs COVID19 worse than flue?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5201\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.4809\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mOf the diagnosed COVID-19 cases, less than 20% are severe.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.4800\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  18\n",
      "Anser: \n",
      "\u001b[33mIt is found that the COVID-19 can be transmitted through droplets, contact, aerosol, etc.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhen the vaccine will be ready?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.7422\n",
      "Paper title: THE CRITICAL ROLE OF NONHUMAN PRIMATES IN MEDICAL RESEARCH White Paper Affiliated Institutions\n",
      "Paper id:    7080c264017e67e26a1e222507314d891b13e3f9\n",
      "Anser size:  9\n",
      "Anser: \n",
      "\u001b[33mAnd is the vaccine likely to be safe?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.7026\n",
      "Paper title: \n",
      "Paper id:    511ac6ac4422cee5357b1b6ff0b59acbfe2402c5\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mIs there a vaccine available or any protective strategy?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.6698\n",
      "Paper title: Q&A: The novel coronavirus outbreak causing COVID-19\n",
      "Paper id:    78360444f3b4540339efc9e7e5f2610a7e46c023\n",
      "Anser size:  11\n",
      "Anser: \n",
      "\u001b[33mWhat is in the pipeline for vaccine development and/or therapeutics?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhats the proteins that consist COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5835\n",
      "Paper title: viruses Extracellular Vesicles and Ebola Virus: A New Mechanism of Immune Evasion\n",
      "Paper id:    dcd6d893dd1fcb8bd484691ccdb9043f3b2d9934\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mIs there one main form or multiple for each of these proteins?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5798\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5630\n",
      "Paper title: ADAM17-dependent signaling is required for oncogenic human papillomavirus entry platform assembly\n",
      "Paper id:    73a2171fc7716dd485b9c3348902a7672213fc0a\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mWhat may be the type of released biomolecules or signaling pathway?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhats the symptoms of COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6368\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mWhat is the percentage of COVID-19 patients have been infected with SARS and produced antibodies?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6033\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5969\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mOf the diagnosed COVID-19 cases, less than 20% are severe.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow can I prevent COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5805\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mreported that people who have not been exposed to SARS-CoV-2 are all susceptible to COVID-19  .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5470\n",
      "Paper title: Early epidemiological analysis of the coronavirus disease 2019 outbreak based on crowdsourced data: a population- level observational study\n",
      "Paper id:    ccc1cedafbb30ee3184f9fc7999f4aa457805cab\n",
      "Anser size:  17\n",
      "Anser: \n",
      "\u001b[33m23 Given the progression of the COVID-19 outbreak, such a long delay would be counterproductive.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5321\n",
      "Paper title: Self-Interest versus Group-Interest in Antiviral Control\n",
      "Paper id:    26670350d15f4a1f0a7072ad985d98b21c28fd5a\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mWhen will a prophylactic antiviral control program be able to prevent an epidemic?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat treatments are available for COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5913\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5875\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mWhat is the percentage of COVID-19 patients have been infected with SARS and produced antibodies?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5633\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mreported that people who have not been exposed to SARS-CoV-2 are all susceptible to COVID-19  .\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mIs hand sanitizer effective against COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5264\n",
      "Paper title: The Common Missed Handwashing Instances and Areas after 15 Years of Hand-Hygiene Education\n",
      "Paper id:    43f3b3e2af86af484ab7aae55ad9b2354e4a14eb\n",
      "Anser size:  9\n",
      "Anser: \n",
      "\u001b[33mthe effectiveness of handwashing and alcohol-based hand sanitizers.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5263\n",
      "Paper title: \n",
      "Paper id:    7461fe0adbb9a865f8a79e994c6cd8b8ebdd4e78\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mIs a mandatory OPV enforced by criminal sanctions the least autonomy-infringing intervention?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5039\n",
      "Paper title: Respiratory viral infections in children with asthma: do they matter and can we prevent them?\n",
      "Paper id:    bbc2824ce7dff3d23d060b7abbe96cba28095fb8\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mThe use of alcohol-based hand sanitizers is also effective [54, 55] .\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mAm I at risk for serious complications from COVID-19 if I smoke cigarettes?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6334\n",
      "Paper title: Influenzanet Agreed Questionnaire 2012\n",
      "Paper id:    c682f8e05b7363e2407b3055826f6c93b6c1f3ad\n",
      "Anser size:  5\n",
      "Anser: \n",
      "\u001b[33mDo you smoke tobacco?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6249\n",
      "Paper title: \n",
      "Paper id:    22c4eb219966e3ab37726362c48b0d8f918d4b52\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mAre you daily exposed to tobacco smoke from others in your home?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5353\n",
      "Paper title: Near-Patient Sampling to Assist Infection Control-A Case Report and Discussion\n",
      "Paper id:    2c798963aaa2245c095624af78d5638bebf60c2f\n",
      "Anser size:  22\n",
      "Anser: \n",
      "\u001b[33mDoes this level of RSV RNA in the air pose a risk to patients, staff or visitors in the vicinity?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mAre there any FDA-approved drugs (medicines) for COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5759\n",
      "Paper title: viruses Who Regulates Whom? An Overview of RNA Granules and Viral Infections\n",
      "Paper id:    e3fe1b272c977754c55719ab93e90ecb0b9472d5\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mFor example, could these mechanisms be targets of new antiviral drugs?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5478\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mWhat is the percentage of COVID-19 patients have been infected with SARS and produced antibodies?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5411\n",
      "Paper title: Self-Interest versus Group-Interest in Antiviral Control\n",
      "Paper id:    26670350d15f4a1f0a7072ad985d98b21c28fd5a\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mWhen will a prophylactic antiviral control program be able to prevent an epidemic?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow are people tested?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.7889\n",
      "Paper title: Ecohealth research in Southeast Asia: past, present and the way forward\n",
      "Paper id:    1495c1fa93db3b9a5d12b5ae15ff0c8639b83452\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mHow can these be tested in practice?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5915\n",
      "Paper title: A Focused Ethnographic Study of Alberta Cattle Veterinarians' Decision Making about Diagnostic Laboratory Submissions and Perceptions of Surveillance Programs\n",
      "Paper id:    721f194d7cd4953a229e067f3e1b79ec2cf22aca\n",
      "Anser size:  9\n",
      "Anser: \n",
      "\u001b[33mWhat kind of value does laboratory testing provide?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5746\n",
      "Paper title: Protocol (version 4 3/7/2009) 1. Title: A PRImary care trial of a website based Infection control intervention to Modify Influenza-like illness and respiratory infection Transmission (1.2 Acronym PRIMIT trial.) 2. 1\n",
      "Paper id:    1575931c6fe7aea67ea3ced0ebe1df4d2604864c\n",
      "Anser size:  11\n",
      "Anser: \n",
      "\u001b[33m2.6 How will the results of this trial be used?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhy is the disease being called coronavirus disease 2019, COVID-19?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.7279\n",
      "Paper title: Cell Discovery Phase-adjusted estimation of the number of Coronavirus Disease 2019 cases in Wuhan, China\n",
      "Paper id:    6abb30ae61aa5e41f16a28b9437940d5d76d745b\n",
      "Anser size:  20\n",
      "Anser: \n",
      "\u001b[33mWorld Health Organization (WHO) now has named the disease Coronavirus Disease 2019 (COVID- 19) 3 .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6803\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5758\n",
      "Paper title: \n",
      "Paper id:    ac37ba61c91bb6939a507dbf4efe2119ddafeb9c\n",
      "Anser size:  17\n",
      "Anser: \n",
      "\u001b[33mWe report two cases of coronavirus disease 2019 in travellers from Wuhan, China to Thailand.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mAm I at risk for COVID-19 from mail, packages, or products?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5496\n",
      "Paper title: Near-Patient Sampling to Assist Infection Control-A Case Report and Discussion\n",
      "Paper id:    2c798963aaa2245c095624af78d5638bebf60c2f\n",
      "Anser size:  22\n",
      "Anser: \n",
      "\u001b[33mDoes this level of RSV RNA in the air pose a risk to patients, staff or visitors in the vicinity?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.4974\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  18\n",
      "Anser: \n",
      "\u001b[33mIt is found that the COVID-19 can be transmitted through droplets, contact, aerosol, etc.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.4962\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mreported that people who have not been exposed to SARS-CoV-2 are all susceptible to COVID-19  .\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is community spread?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.4822\n",
      "Paper title: Public Health Network Structure and Collaboration Effectiveness during the 2015 MERS Outbreak in South Korea: An Institutional Collective Action Framework\n",
      "Paper id:    afa67c34e52aa31421d540d1ea7484f52b3318e2\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mHow does effective dissemination of information affect interorganizational collaboration?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.4817\n",
      "Paper title: Ecohealth research in Southeast Asia: past, present and the way forward\n",
      "Paper id:    1495c1fa93db3b9a5d12b5ae15ff0c8639b83452\n",
      "Anser size:  6\n",
      "Anser: \n",
      "\u001b[33mWhat are the marketing networks?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.4514\n",
      "Paper title: Improving the use of research evidence in guideline development: 13. Applicability, transferability and adaptation\n",
      "Paper id:    c878b060760db3b59ae0129a5de677e56a30866e\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mWhat should be done centrally and locally?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow can I protect myself?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.5298\n",
      "Paper title: Knowledge and attitudes of university students toward pandemic influenza: a cross-sectional study from Turkey\n",
      "Paper id:    545def8771357b4cb2875f5795a0760e97534cc9\n",
      "Anser size:  17\n",
      "Anser: \n",
      "\u001b[33m\"What do you do for protection\" was the survey's one of open-ended questions.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.5114\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mHow should I do to avoid infection?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5107\n",
      "Paper title: Social Consequences of Ebola Containment Measures in Liberia\n",
      "Paper id:    3c3068bf6e652c02d85923d2438ad4c84d53071a\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mHow do people perceive interventions to protect their health?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is a novel coronavirus?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.6385\n",
      "Paper title: Tropical Medicine and Infectious Disease Potential Intermediate Hosts for Coronavirus Transmission: No Evidence of Clade 2c Coronaviruses in Domestic Livestock from Ghana\n",
      "Paper id:    95cc317541d97e3dbaa1662894fdbed842098910\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mcoronavirus.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6272\n",
      "Paper title: Could human coronavirus OC43 have co-evolved with early humans?\n",
      "Paper id:    123db7e17bb17a0d4ff1e7ef87cb629d6ac78125\n",
      "Anser size:  3\n",
      "Anser: \n",
      "\u001b[33mHuman coronavirus evolution\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.6269\n",
      "Paper title: Population genetics, community of parasites, and resistance to rodenticides in an urban brown rat (Rattus norvegicus) population\n",
      "Paper id:    c8d60caf44017989b3b9633350fc1d2efda570a5\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mCoronavirus.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWas Harry Potter a good magician?\u001b[0m\n",
      "\n",
      "->  1\n",
      "Score: 0.3480\n",
      "Paper title: Reassuring and managing patients with concerns about swine flu: Qualitative interviews with callers to NHS Direct\n",
      "Paper id:    6fc8c1b4cfbd790cbe02c825fd6f9181b1fc6645\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mDid you find that reassuring or useful?\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.3171\n",
      "Paper title: Computer aided selection of candidate vaccine antigens\n",
      "Paper id:    e7a8dfcde06b70cb86b30074d9008635252d6301\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mAre apparently high-scoring matches real or an artefact of search methodology?\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.3071\n",
      "Paper title: Effect of cinnamon water extract on monocyte-to- macrophage differentiation and scavenger receptor activity\n",
      "Paper id:    b8660d5f12c9a6a85b73207bc61291535c2700ce\n",
      "Anser size:  17\n",
      "Anser: \n",
      "\u001b[33mCinnamon is a popular spice around the world and is one of the oldest medicinal herbs.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "  top_k = 3\n",
    "  evaluation(top_k,myquery_list,corpus_embeddings,myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zd29996wfURy"
   },
   "source": [
    "# Overall results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w5o0GT6OglvS"
   },
   "source": [
    "## 6000 papers with paraphrase-distilroberta-base-v1 model and no summarization \n",
    "---\n",
    "\n",
    "### Time needed for creating the embeddings: \n",
    "- CPU times: \n",
    "  - user 17min 51s\n",
    "  - sys: 11min 38s\n",
    "  - total: 29min 29s\n",
    "- Wall time: 29min 11s\n",
    "\n",
    "### Remarks\n",
    "Some questions have been anwered but I see a lot of small answers that I shouldn't have in the corpus and certainly don't answer the questions. Eliminating small sentences might have better results.\n",
    "\n",
    "__Top-k__ \n",
    "\n",
    "I can obviously see that there are some answers in the top-3 that fit better than the first and best answer.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tYoPMvbLlWr0"
   },
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 484
    },
    "id": "9JWa2KoLfoo-",
    "outputId": "c99c7de6-a6b9-4078-958f-e13278d2b496"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes.</td>\n",
       "      <td>Avian coronaviruses.</td>\n",
       "      <td>0.617210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>Recently, in December 2019, an outbreak of unusual pneumonia caused by unknown infection was reported in Wuhan, China  .</td>\n",
       "      <td>0.541798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.</td>\n",
       "      <td>Coronavirus.</td>\n",
       "      <td>0.675533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.</td>\n",
       "      <td>The first three imported cases of COVID-19 in France, the first ones in Europe, were diagnosed 14 days later, on 24 January.</td>\n",
       "      <td>0.515843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.</td>\n",
       "      <td>The SARS outbreak of 2002-2003 was caused by SARS-CoV, a novel coronavirus.</td>\n",
       "      <td>0.615217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.</td>\n",
       "      <td>The international expansion of COVID-19 cases has led to widespread adoption of symptom and risk screening measures, in travel-associated and other contexts, and programs may still be adopted or expanded as source epidemics of COVID-19 emerge in new geographic areas.</td>\n",
       "      <td>0.542352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>No identified contact of the three cases has been confirmed with COVID-19.</td>\n",
       "      <td>0.530918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.</td>\n",
       "      <td>This may hint that coronavirus does not spread so widely in humans.</td>\n",
       "      <td>0.670806</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...          0.617210\n",
       "1  What was discovered in Wuhuan in December 2019?  ...          0.541798\n",
       "2                What is Coronovirus Disease 2019?  ...          0.675533\n",
       "3                                What is COVID-19?  ...          0.515843\n",
       "4                     What is caused by SARS-COV2?  ...          0.615217\n",
       "5                          How is COVID-19 spread?  ...          0.542352\n",
       "6                   Where was COVID-19 discovered?  ...          0.530918\n",
       "7                     How does coronavirus spread?  ...          0.670806\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 743
    },
    "id": "_0_wfnalfpGL",
    "outputId": "7774c2d3-ca44-4e79-8a00-d95887a0e1c9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How long can the coronavirus survive on surfaces?</td>\n",
       "      <td>These data imply that the coronavirus can persist in its host for at least the duration of hibernation, particularly as nucleotide variability among the detected coronavirus isolates showed that spread of coronavirus among bats within a chamber was unlikely 32 .</td>\n",
       "      <td>0.647764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What means COVID-19?</td>\n",
       "      <td>In the context of COVID-19, we consider both growing and stable epidemic scenarios, but place greater emphasis on the realistic assumption that the COVID-19 epidemic is still growing.</td>\n",
       "      <td>0.522382</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Is COVID19 worse than flue?</td>\n",
       "      <td>The first three imported cases of COVID-19 in France, the first ones in Europe, were diagnosed 14 days later, on 24 January.</td>\n",
       "      <td>0.476099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>When the vaccine will be ready?</td>\n",
       "      <td>It is unknown if the vaccines will work.</td>\n",
       "      <td>0.673893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Whats the proteins that consist COVID-19?</td>\n",
       "      <td>Is there one main form or multiple for each of these proteins?</td>\n",
       "      <td>0.583473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Whats the symptoms of COVID-19?</td>\n",
       "      <td>The true fraction of subclinical COVID-19 cases remains unknown, but anecdotally, many lab-confirmed COVID-19 cases have not shown detectable symptoms on diagnosis (Hoehl et al., 2020; Nishiura et al., 2020; .</td>\n",
       "      <td>0.630991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>How can I prevent COVID-19?</td>\n",
       "      <td>At the same time, there is great concern about potential public health consequences if COVID-19 spreads to developing countries that lack health infrastructure and resources to combat it effectively (de Salazar et al., 2020) .</td>\n",
       "      <td>0.521944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What treatments are available for COVID-19?</td>\n",
       "      <td>The true fraction of subclinical COVID-19 cases remains unknown, but anecdotally, many lab-confirmed COVID-19 cases have not shown detectable symptoms on diagnosis (Hoehl et al., 2020; Nishiura et al., 2020; .</td>\n",
       "      <td>0.600732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Is hand sanitizer effective against COVID-19?</td>\n",
       "      <td>Is a mandatory OPV enforced by criminal sanctions the least autonomy-infringing intervention?</td>\n",
       "      <td>0.526254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Am I at risk for serious complications from COVID-19 if I smoke cigarettes?</td>\n",
       "      <td>Does this level of RSV RNA in the air pose a risk to patients, staff or visitors in the vicinity?</td>\n",
       "      <td>0.535259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Are there any FDA-approved drugs (medicines) for COVID-19?</td>\n",
       "      <td>As a drug that is contained in a plant or its fruit, should it be evaluated, authorized, and supervised by Public Health Institutes or a similar Human Health Organization in each country?</td>\n",
       "      <td>0.598755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>How are people tested?</td>\n",
       "      <td>were tested.</td>\n",
       "      <td>0.723114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Why is the disease being called coronavirus disease 2019, COVID-19?</td>\n",
       "      <td>and \"is the severity of flavivirus infection associated with higher prevalence of comorbidities?\"</td>\n",
       "      <td>0.569618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Am I at risk for COVID-19 from mail, packages, or products?</td>\n",
       "      <td>Does this level of RSV RNA in the air pose a risk to patients, staff or visitors in the vicinity?</td>\n",
       "      <td>0.549572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>What is community spread?</td>\n",
       "      <td>By community similarity, we mean not just the number of communities but the composition of those communities as well.</td>\n",
       "      <td>0.492949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>How can I protect myself?</td>\n",
       "      <td>How do people perceive interventions to protect their health?</td>\n",
       "      <td>0.510747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>What is a novel coronavirus?</td>\n",
       "      <td>These viruses are coronavirus.</td>\n",
       "      <td>0.660625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Was Harry Potter a good magician?</td>\n",
       "      <td>Is it possible that herbs and TCM formulas can be effective in this low concentration of bioactive compounds?</td>\n",
       "      <td>0.356533</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          Query  ... Cosine_similarity\n",
       "0                             How long can the coronavirus survive on surfaces?  ...          0.647764\n",
       "1                                                          What means COVID-19?  ...          0.522382\n",
       "2                                                   Is COVID19 worse than flue?  ...          0.476099\n",
       "3                                               When the vaccine will be ready?  ...          0.673893\n",
       "4                                     Whats the proteins that consist COVID-19?  ...          0.583473\n",
       "5                                               Whats the symptoms of COVID-19?  ...          0.630991\n",
       "6                                                   How can I prevent COVID-19?  ...          0.521944\n",
       "7                                   What treatments are available for COVID-19?  ...          0.600732\n",
       "8                                 Is hand sanitizer effective against COVID-19?  ...          0.526254\n",
       "9   Am I at risk for serious complications from COVID-19 if I smoke cigarettes?  ...          0.535259\n",
       "10                   Are there any FDA-approved drugs (medicines) for COVID-19?  ...          0.598755\n",
       "11                                                       How are people tested?  ...          0.723114\n",
       "12          Why is the disease being called coronavirus disease 2019, COVID-19?  ...          0.569618\n",
       "13                  Am I at risk for COVID-19 from mail, packages, or products?  ...          0.549572\n",
       "14                                                    What is community spread?  ...          0.492949\n",
       "15                                                    How can I protect myself?  ...          0.510747\n",
       "16                                                 What is a novel coronavirus?  ...          0.660625\n",
       "17                                            Was Harry Potter a good magician?  ...          0.356533\n",
       "\n",
       "[18 rows x 3 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i_BGu6SMjNTz"
   },
   "source": [
    "## 9000 papers with paraphrase-distilroberta-base-v1 model and no summarization \n",
    "---\n",
    "\n",
    "Session crashed after using all available RAM, this happened every time I tried to run it with all the papers and no summarization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yvuJfjaQqYU_"
   },
   "source": [
    "## 6000 papers with paraphrase-distilroberta-base-v1 model and summarization \n",
    "---\n",
    "\n",
    "### Time needed for creating the embeddings: \n",
    "- CPU times: \n",
    "  - user 1min 12s\n",
    "  - sys: 25.4 s\n",
    "  - total: 1min 37s\n",
    "- Wall time: 1min 42s\n",
    "\n",
    "### Remarks\n",
    "Same not good results. I also notice from these results that there are lots of answers that are questions from the papers. In my understanding, these answers occur due to the questionmark. I thought of two ways of improving this problem but I had no time re running all these notebooks (due to CUDA mostly):\n",
    "\n",
    "- Either by removing the questionmark in pre-process. As fas as I searched it this isn't a very good solution for tackling this problem.\n",
    "- Either by removing the all the sentences with a questionmark (questions) in pre-process. This I believe whould be the best solution, but I hadn't time to check it.\n",
    "\n",
    "__Top-k__\n",
    "\n",
    "Again lots of good answers were in the top-2 or top-3.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zl1L-QtfqYVW"
   },
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 484
    },
    "id": "4BVtzGYLqYVY",
    "outputId": "c99c7de6-a6b9-4078-958f-e13278d2b496"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes.</td>\n",
       "      <td>Avian coronaviruses.</td>\n",
       "      <td>0.617210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>Recently, in December 2019, an outbreak of unusual pneumonia caused by unknown infection was reported in Wuhan, China  .</td>\n",
       "      <td>0.541798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.</td>\n",
       "      <td>Coronavirus.</td>\n",
       "      <td>0.675533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.</td>\n",
       "      <td>The first three imported cases of COVID-19 in France, the first ones in Europe, were diagnosed 14 days later, on 24 January.</td>\n",
       "      <td>0.515843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.</td>\n",
       "      <td>The SARS outbreak of 2002-2003 was caused by SARS-CoV, a novel coronavirus.</td>\n",
       "      <td>0.615217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.</td>\n",
       "      <td>The international expansion of COVID-19 cases has led to widespread adoption of symptom and risk screening measures, in travel-associated and other contexts, and programs may still be adopted or expanded as source epidemics of COVID-19 emerge in new geographic areas.</td>\n",
       "      <td>0.542352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>No identified contact of the three cases has been confirmed with COVID-19.</td>\n",
       "      <td>0.530918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.</td>\n",
       "      <td>This may hint that coronavirus does not spread so widely in humans.</td>\n",
       "      <td>0.670806</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...          0.617210\n",
       "1  What was discovered in Wuhuan in December 2019?  ...          0.541798\n",
       "2                What is Coronovirus Disease 2019?  ...          0.675533\n",
       "3                                What is COVID-19?  ...          0.515843\n",
       "4                     What is caused by SARS-COV2?  ...          0.615217\n",
       "5                          How is COVID-19 spread?  ...          0.542352\n",
       "6                   Where was COVID-19 discovered?  ...          0.530918\n",
       "7                     How does coronavirus spread?  ...          0.670806\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 607
    },
    "id": "s2SITV9WqYVb",
    "outputId": "06fce453-0822-4423-a739-8b8df5f85ba0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How long can the coronavirus survive on surfaces?</td>\n",
       "      <td>This may hint that coronavirus does not spread so widely in humans.</td>\n",
       "      <td>0.597280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What means COVID-19?</td>\n",
       "      <td>established for HCoV-OC43 ns12.9 (25), HCoV-229E 4a (24) , and PEDV 3 (26) .</td>\n",
       "      <td>0.465204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Is COVID19 worse than flue?</td>\n",
       "      <td>So why do Zika and Dengue NS2B have such radical differences in conformations and dynamics?</td>\n",
       "      <td>0.443893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>When the vaccine will be ready?</td>\n",
       "      <td>And is the vaccine likely to be safe?</td>\n",
       "      <td>0.742244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Whats the proteins that consist COVID-19?</td>\n",
       "      <td>Is there one main form or multiple for each of these proteins?</td>\n",
       "      <td>0.583473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Whats the symptoms of COVID-19?</td>\n",
       "      <td>What is the clinical profile of co-infected patients?</td>\n",
       "      <td>0.576397</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>How can I prevent COVID-19?</td>\n",
       "      <td>So how can the COPII machinery manage to transport large viral agglomerates?</td>\n",
       "      <td>0.480076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What treatments are available for COVID-19?</td>\n",
       "      <td>What is the clinical profile of co-infected patients?</td>\n",
       "      <td>0.502994</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Is hand sanitizer effective against COVID-19?</td>\n",
       "      <td>the effectiveness of handwashing and alcohol-based hand sanitizers.</td>\n",
       "      <td>0.526350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Am I at risk for serious complications from COVID-19 if I smoke cigarettes?</td>\n",
       "      <td>Do you smoke tobacco?</td>\n",
       "      <td>0.633407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Are there any FDA-approved drugs (medicines) for COVID-19?</td>\n",
       "      <td>For example, could these mechanisms be targets of new antiviral drugs?</td>\n",
       "      <td>0.575899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>How are people tested?</td>\n",
       "      <td>How can these be tested in practice?</td>\n",
       "      <td>0.788895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Why is the disease being called coronavirus disease 2019, COVID-19?</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.680306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Am I at risk for COVID-19 from mail, packages, or products?</td>\n",
       "      <td>Does this level of RSV RNA in the air pose a risk to patients, staff or visitors in the vicinity?</td>\n",
       "      <td>0.549572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>What is community spread?</td>\n",
       "      <td>How does effective dissemination of information affect interorganizational collaboration?</td>\n",
       "      <td>0.482189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>How can I protect myself?</td>\n",
       "      <td>How do people perceive interventions to protect their health?</td>\n",
       "      <td>0.510747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>What is a novel coronavirus?</td>\n",
       "      <td>coronavirus.</td>\n",
       "      <td>0.638488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Was Harry Potter a good magician?</td>\n",
       "      <td>Royal jelly is a protein-rich secretion that serves as food for larvae and adult queen honey bees.</td>\n",
       "      <td>0.302791</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          Query  ... Cosine_similarity\n",
       "0                             How long can the coronavirus survive on surfaces?  ...          0.597280\n",
       "1                                                          What means COVID-19?  ...          0.465204\n",
       "2                                                   Is COVID19 worse than flue?  ...          0.443893\n",
       "3                                               When the vaccine will be ready?  ...          0.742244\n",
       "4                                     Whats the proteins that consist COVID-19?  ...          0.583473\n",
       "5                                               Whats the symptoms of COVID-19?  ...          0.576397\n",
       "6                                                   How can I prevent COVID-19?  ...          0.480076\n",
       "7                                   What treatments are available for COVID-19?  ...          0.502994\n",
       "8                                 Is hand sanitizer effective against COVID-19?  ...          0.526350\n",
       "9   Am I at risk for serious complications from COVID-19 if I smoke cigarettes?  ...          0.633407\n",
       "10                   Are there any FDA-approved drugs (medicines) for COVID-19?  ...          0.575899\n",
       "11                                                       How are people tested?  ...          0.788895\n",
       "12          Why is the disease being called coronavirus disease 2019, COVID-19?  ...          0.680306\n",
       "13                  Am I at risk for COVID-19 from mail, packages, or products?  ...          0.549572\n",
       "14                                                    What is community spread?  ...          0.482189\n",
       "15                                                    How can I protect myself?  ...          0.510747\n",
       "16                                                 What is a novel coronavirus?  ...          0.638488\n",
       "17                                            Was Harry Potter a good magician?  ...          0.302791\n",
       "\n",
       "[18 rows x 3 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TaV_Hi_9qefX"
   },
   "source": [
    "## 9000 papers with paraphrase-distilroberta-base-v1 model and summarization \n",
    "---\n",
    "\n",
    "CPU times: user , sys: , total: \n",
    "Wall time: \n",
    "\n",
    "### Time needed for creating the embeddings: \n",
    "- CPU times: \n",
    "  - user 1min 45s\n",
    "  - sys: 35.7 s\n",
    "  - total: 2min 21s\n",
    "- Wall time: 2min 17s\n",
    "\n",
    "### Remarks\n",
    "Despite increasing the corpus (summarized) with 3000 papers more, I see worse results from before. Same problem, answering with questions or irrelevant answers. Can't understand what exactly happens as this is the third time I fix pre-process and re-run my models.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DGMbvlo-qefZ"
   },
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 416
    },
    "id": "_PLXWjj0qefa",
    "outputId": "98cacf1b-8dc4-48b0-f363-6d03d1faa4f1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes.</td>\n",
       "      <td>Coronaviruses: what are they?</td>\n",
       "      <td>0.632730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>Nine of the first 11 U.S. 2019-nCoV patients were exposed in Wuhan, China.</td>\n",
       "      <td>0.479242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.</td>\n",
       "      <td>Coronavirus.</td>\n",
       "      <td>0.675533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.</td>\n",
       "      <td>This is particularly true for the COVID-19.</td>\n",
       "      <td>0.665808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.</td>\n",
       "      <td>Is there an effective specific anti-SARS-CoV-2 solution?</td>\n",
       "      <td>0.661237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.</td>\n",
       "      <td>The COVID-19 has then rapidly spread to all over China and the world.</td>\n",
       "      <td>0.625521</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>The main dataset of this study is COVID-19 dataset.</td>\n",
       "      <td>0.595511</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.</td>\n",
       "      <td>The pathogenesis of feline coronavirus infection is unclear.</td>\n",
       "      <td>0.679283</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...          0.632730\n",
       "1  What was discovered in Wuhuan in December 2019?  ...          0.479242\n",
       "2                What is Coronovirus Disease 2019?  ...          0.675533\n",
       "3                                What is COVID-19?  ...          0.665808\n",
       "4                     What is caused by SARS-COV2?  ...          0.661237\n",
       "5                          How is COVID-19 spread?  ...          0.625521\n",
       "6                   Where was COVID-19 discovered?  ...          0.595511\n",
       "7                     How does coronavirus spread?  ...          0.679283\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 607
    },
    "id": "uV8T__Ulqefc",
    "outputId": "924f2784-ad72-44e4-aefd-e68da1cde897"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How long can the coronavirus survive on surfaces?</td>\n",
       "      <td>The pathogenesis of feline coronavirus infection is unclear.</td>\n",
       "      <td>0.641915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What means COVID-19?</td>\n",
       "      <td>This is particularly true for the COVID-19.</td>\n",
       "      <td>0.675011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Is COVID19 worse than flue?</td>\n",
       "      <td>This is particularly true for the COVID-19.</td>\n",
       "      <td>0.520135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>When the vaccine will be ready?</td>\n",
       "      <td>And is the vaccine likely to be safe?</td>\n",
       "      <td>0.742244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Whats the proteins that consist COVID-19?</td>\n",
       "      <td>Is there one main form or multiple for each of these proteins?</td>\n",
       "      <td>0.583473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Whats the symptoms of COVID-19?</td>\n",
       "      <td>What is the percentage of COVID-19 patients have been infected with SARS and produced antibodies?</td>\n",
       "      <td>0.636812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>How can I prevent COVID-19?</td>\n",
       "      <td>reported that people who have not been exposed to SARS-CoV-2 are all susceptible to COVID-19  .</td>\n",
       "      <td>0.580536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What treatments are available for COVID-19?</td>\n",
       "      <td>It remains to be seen if this will be the case for COVID-19 as well.</td>\n",
       "      <td>0.591324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Is hand sanitizer effective against COVID-19?</td>\n",
       "      <td>the effectiveness of handwashing and alcohol-based hand sanitizers.</td>\n",
       "      <td>0.526350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Am I at risk for serious complications from COVID-19 if I smoke cigarettes?</td>\n",
       "      <td>Do you smoke tobacco?</td>\n",
       "      <td>0.633407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Are there any FDA-approved drugs (medicines) for COVID-19?</td>\n",
       "      <td>For example, could these mechanisms be targets of new antiviral drugs?</td>\n",
       "      <td>0.575899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>How are people tested?</td>\n",
       "      <td>How can these be tested in practice?</td>\n",
       "      <td>0.788895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Why is the disease being called coronavirus disease 2019, COVID-19?</td>\n",
       "      <td>World Health Organization (WHO) now has named the disease Coronavirus Disease 2019 (COVID- 19) 3 .</td>\n",
       "      <td>0.727911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Am I at risk for COVID-19 from mail, packages, or products?</td>\n",
       "      <td>Does this level of RSV RNA in the air pose a risk to patients, staff or visitors in the vicinity?</td>\n",
       "      <td>0.549572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>What is community spread?</td>\n",
       "      <td>How does effective dissemination of information affect interorganizational collaboration?</td>\n",
       "      <td>0.482189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>How can I protect myself?</td>\n",
       "      <td>\"What do you do for protection\" was the survey's one of open-ended questions.</td>\n",
       "      <td>0.529803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>What is a novel coronavirus?</td>\n",
       "      <td>coronavirus.</td>\n",
       "      <td>0.638488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Was Harry Potter a good magician?</td>\n",
       "      <td>Did you find that reassuring or useful?</td>\n",
       "      <td>0.347960</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          Query  ... Cosine_similarity\n",
       "0                             How long can the coronavirus survive on surfaces?  ...          0.641915\n",
       "1                                                          What means COVID-19?  ...          0.675011\n",
       "2                                                   Is COVID19 worse than flue?  ...          0.520135\n",
       "3                                               When the vaccine will be ready?  ...          0.742244\n",
       "4                                     Whats the proteins that consist COVID-19?  ...          0.583473\n",
       "5                                               Whats the symptoms of COVID-19?  ...          0.636812\n",
       "6                                                   How can I prevent COVID-19?  ...          0.580536\n",
       "7                                   What treatments are available for COVID-19?  ...          0.591324\n",
       "8                                 Is hand sanitizer effective against COVID-19?  ...          0.526350\n",
       "9   Am I at risk for serious complications from COVID-19 if I smoke cigarettes?  ...          0.633407\n",
       "10                   Are there any FDA-approved drugs (medicines) for COVID-19?  ...          0.575899\n",
       "11                                                       How are people tested?  ...          0.788895\n",
       "12          Why is the disease being called coronavirus disease 2019, COVID-19?  ...          0.727911\n",
       "13                  Am I at risk for COVID-19 from mail, packages, or products?  ...          0.549572\n",
       "14                                                    What is community spread?  ...          0.482189\n",
       "15                                                    How can I protect myself?  ...          0.529803\n",
       "16                                                 What is a novel coronavirus?  ...          0.638488\n",
       "17                                            Was Harry Potter a good magician?  ...          0.347960\n",
       "\n",
       "[18 rows x 3 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zmy8pUVD7Drn"
   },
   "source": [
    "# References\n",
    "\n",
    "[1] https://www.sbert.net/examples/applications/semantic-search/README.html\n",
    "\n",
    "[2] https://colab.research.google.com/drive/12cn5Oo0v3HfQQ8Tv6-ukgxXSmT3zl35A?usp=sharing"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "SBERT_CORD19_QA_Roberta.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "46c9e683d3d74f95873119c1222986d9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5559cb96327544a3b40ae7a0aa5ff6f3",
       "IPY_MODEL_b557158e8e284a76b5080c5155354bb7"
      ],
      "layout": "IPY_MODEL_7f45b4f8e3aa47b0b8ecbccfd2f56c75"
     }
    },
    "47bce05c5be7463fa80c63547f647939": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5559cb96327544a3b40ae7a0aa5ff6f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Batches: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_627962abe74d4b519229906cb57e7ef3",
      "max": 6793,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_dc7eec0dc4734d358a5f0930b9227e6a",
      "value": 6793
     }
    },
    "627962abe74d4b519229906cb57e7ef3": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "730c3e1d7d8649cc9ee495ad8a6e9996": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7f45b4f8e3aa47b0b8ecbccfd2f56c75": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b557158e8e284a76b5080c5155354bb7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_47bce05c5be7463fa80c63547f647939",
      "placeholder": "​",
      "style": "IPY_MODEL_730c3e1d7d8649cc9ee495ad8a6e9996",
      "value": " 6793/6793 [02:15&lt;00:00, 50.31it/s]"
     }
    },
    "dc7eec0dc4734d358a5f0930b9227e6a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}