SBERT_CORD19_QA_CrossEncoders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T28QEDT-Uqj2"
   },
   "source": [
    "<p align=\"center\">\n",
    " <img src=\"http://www.di.uoa.gr/themes/corporate_lite/logo_el.png\" title=\"Department of Informatics and Telecommunications - University of Athens\"/> </p>\n",
    "\n",
    "---\n",
    "<h1 align=\"center\"> \n",
    "  Artificial Intelligence\n",
    "</h1>\n",
    "<h1 align=\"center\" > \n",
    "  Deep Learning for Natural Language Processing\n",
    "</h1>\n",
    "\n",
    "---\n",
    "<h2 align=\"center\"> \n",
    " <b>Konstantinos Nikoletos</b>\n",
    "</h2>\n",
    "\n",
    "<h3 align=\"center\"> \n",
    " <b>Winter 2020-2021</b>\n",
    "</h3>\n",
    "\n",
    "\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wk9SOEk6mCXZ"
   },
   "source": [
    "\n",
    "### __Task__ \n",
    "This exercise is about developing a document retrieval system to return titles of scientific\n",
    "papers containing the answer to a given user question. You will use the first version of\n",
    "the COVID-19 Open Research Dataset (CORD-19) in your work (articles in the folder\n",
    "comm use subset).\n",
    "\n",
    "\n",
    "For example, for the question “What are the coronaviruses?”, your system can return the\n",
    "paper title “Distinct Roles for Sialoside and Protein Receptors in Coronavirus Infection”\n",
    "since this paper contains the answer to the asked question.\n",
    "\n",
    "\n",
    "To achieve the goal of this exercise, you will need first to read the paper Sentence-BERT:\n",
    "Sentence Embeddings using Siamese BERT-Networks, in order to understand how you\n",
    "can create sentence embeddings. In the related work of this paper, you will also find other\n",
    "approaches for developing your model. For example, you can using Glove embeddings,\n",
    "etc. In this link, you can find the extended versions of this dataset to test your model, if\n",
    "you want. You are required to:\n",
    "\n",
    "\n",
    "<ol type=\"a\">\n",
    "  <li>Preprocess the provided dataset. You will decide which data of each paper is useful\n",
    "to your model in order to create the appropriate embeddings. You need to explain\n",
    "your decisions.</li>\n",
    "  <li>Implement at least 2 different sentence embedding approaches (see the related work\n",
    "of the Sentence-BERT paper), in order for your model to retrieve the titles of the\n",
    "papers related to a given question.</li>\n",
    "  <li>Compare your 2 models based on at least 2 different criteria of your choice. Explain\n",
    "why you selected these criteria, your implementation choices, and the results. Some\n",
    "questions you can pose are included here. You will need to provide the extra questions\n",
    "you posed to your model and the results of all the questions as well.</li>\n",
    "</ol>\n",
    "\n",
    "### __Notebook__ \n",
    "\n",
    "\n",
    "Same implementation as Sentence Bert notebook but with adding CrossEncoders that I read that they perform even better \n",
    "\n",
    "\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6OfAccyufasd"
   },
   "source": [
    "__Import__ of essential libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dVSHtKTrfGiV",
    "outputId": "2a3dc1ee-44a0-4847-9dcd-6421135567e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /root/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd \n",
    "import sys # only needed to determine Python version number\n",
    "import matplotlib # only needed to determine Matplotlib version \n",
    "import nltk\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import pprint\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torchtext import data\n",
    "import logging\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('averaged_perceptron_tagger')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vC9DWskzConr"
   },
   "source": [
    "Selecting device (GPU - CUDA if available)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NiRkpLhiCntl",
    "outputId": "814d2a4b-8bef-4a42-d576-f48bed8d5809"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on GPU.\n"
     ]
    }
   ],
   "source": [
    "# First checking if GPU is available\n",
    "train_on_gpu=torch.cuda.is_available()\n",
    "\n",
    "if(train_on_gpu):\n",
    "    print('Training on GPU.')\n",
    "else:\n",
    "    print('No GPU available, training on CPU.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cvK_TWa2YNg5"
   },
   "source": [
    "# Loading data\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "D6nnb0BFMZDx",
    "outputId": "a0f66d72-1f3d-4264-ca71-b576455d2ff1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "# Opening data file\n",
    "import io\n",
    "from google.colab import drive\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import json\n",
    "\n",
    "drive.mount('/content/drive',force_remount=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wih8b1VKVeS2"
   },
   "source": [
    "Loading the dictionary if it has been created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "a8m_oar6ZzH3"
   },
   "outputs": [],
   "source": [
    "#@title Select number of papers that will be feeded in the model { vertical-output: true, display-mode: \"both\" }\n",
    "number_of_papers = \"9000\" #@param [\"1000\",\"3000\", \"6000\",\"9000\"]\n",
    "import pickle\n",
    "\n",
    "CORD19_Dataframe = r\"/content/drive/My Drive/AI_4/CORD19_SentenceMap_\"+number_of_papers+\".pkl\"\n",
    "with open(CORD19_Dataframe, 'rb') as drivef:\n",
    "  CORD19Dictionary = pickle.load(drivef)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2xLN9x3B6KfR"
   },
   "source": [
    "OR the summary of the papers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A2Qk-Pnn6Kfg"
   },
   "outputs": [],
   "source": [
    "#@title Select number of summarized papers that will be feeded in the model { vertical-output: true, display-mode: \"both\" }\n",
    "number_of_papers = \"9000\" #@param [\"1000\", \"3000\", \"6000\", \"9000\"]\n",
    "import pickle\n",
    "\n",
    "CORD19_Dataframe = r\"/content/drive/My Drive/AI_4/CORD19_SentenceMap_Summarized_\"+number_of_papers+\".pkl\"\n",
    "with open(CORD19_Dataframe, 'rb') as drivef:\n",
    "  CORD19Dictionary = pickle.load(drivef)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aJ16If_pdTXV"
   },
   "source": [
    "## Queries\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PdGaXWMdWIdh"
   },
   "outputs": [],
   "source": [
    "query_list = [\n",
    "  'What are the coronoviruses?',\n",
    "  'What was discovered in Wuhuan in December 2019?',\n",
    "  'What is Coronovirus Disease 2019?',\n",
    "  'What is COVID-19?',\n",
    "  'What is caused by SARS-COV2?', 'How is COVID-19 spread?',\n",
    "  'Where was COVID-19 discovered?','How does coronavirus spread?'\n",
    "]\n",
    "\n",
    "proposed_answers = [\n",
    "  'Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes. ',\n",
    "  'In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.',\n",
    "  'Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.',\n",
    "  'COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.',\n",
    "  'Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.', \n",
    "  'First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.',\n",
    "  'In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.',\n",
    "  'The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.'\n",
    "]\n",
    "\n",
    "myquery_list = [\n",
    "  \"How long can the coronavirus survive on surfaces?\",\n",
    "  \"What means COVID-19?\",\n",
    "  \"Is COVID19 worse than flue?\",\n",
    "  \"When the vaccine will be ready?\",\n",
    "  \"Whats the proteins that consist COVID-19?\",\n",
    "  \"Whats the symptoms of COVID-19?\",\n",
    "  \"How can I prevent COVID-19?\",\n",
    "  \"What treatments are available for COVID-19?\",\n",
    "  \"Is hand sanitizer effective against COVID-19?\",\n",
    "  \"Am I at risk for serious complications from COVID-19 if I smoke cigarettes?\",\n",
    "  \"Are there any FDA-approved drugs (medicines) for COVID-19?\",\n",
    "  \"How are people tested?\",\n",
    "  \"Why is the disease being called coronavirus disease 2019, COVID-19?\",\n",
    "  \"Am I at risk for COVID-19 from mail, packages, or products?\",\n",
    "  \"What is community spread?\",\n",
    "  \"How can I protect myself?\",\n",
    "  \"What is a novel coronavirus?\",\n",
    "  \"Was Harry Potter a good magician?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nwPB-YNeW6t2"
   },
   "source": [
    "# Results dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "_kBsV1FlW_4y",
    "outputId": "15c773e6-29cf-4796-92d3-8ee92b973807"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and anim...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emer...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SA...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airb...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...               NaN\n",
       "1  What was discovered in Wuhuan in December 2019?  ...               NaN\n",
       "2                What is Coronovirus Disease 2019?  ...               NaN\n",
       "3                                What is COVID-19?  ...               NaN\n",
       "4                     What is caused by SARS-COV2?  ...               NaN\n",
       "5                          How is COVID-19 spread?  ...               NaN\n",
       "6                   Where was COVID-19 discovered?  ...               NaN\n",
       "7                     How does coronavirus spread?  ...               NaN\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resultsDf = pd.DataFrame(columns=['Number of papers','Embeddings creation time'])\n",
    "\n",
    "queriesDf = pd.DataFrame(columns=['Query','Proposed_answer','Model_answer','Cosine_similarity'])\n",
    "queriesDf['Query'] = query_list\n",
    "queriesDf['Proposed_answer'] = proposed_answers\n",
    "\n",
    "myQueriesDf = pd.DataFrame(columns=['Query','Model_answer','Cosine_similarity'])\n",
    "myQueriesDf['Query'] = myquery_list\n",
    "\n",
    "queriesDf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ODeyHCYINWRR"
   },
   "source": [
    "# SBERT\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "J7g1wtxuwlkx",
    "outputId": "5f474888-d181-423e-d16e-6eac4ccdbcae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: sentence-transformers in /usr/local/lib/python3.6/dist-packages (0.4.1.2)\n",
      "Requirement already satisfied, skipping upgrade: torch>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.7.0+cu101)\n",
      "Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.19.5)\n",
      "Requirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (4.41.1)\n",
      "Requirement already satisfied, skipping upgrade: sentencepiece in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (0.1.95)\n",
      "Requirement already satisfied, skipping upgrade: transformers<5.0.0,>=3.1.0 in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (4.3.2)\n",
      "Requirement already satisfied, skipping upgrade: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (0.22.2.post1)\n",
      "Requirement already satisfied, skipping upgrade: nltk in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (3.2.5)\n",
      "Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.4.1)\n",
      "Requirement already satisfied, skipping upgrade: future in /usr/local/lib/python3.6/dist-packages (from torch>=1.6.0->sentence-transformers) (0.16.0)\n",
      "Requirement already satisfied, skipping upgrade: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch>=1.6.0->sentence-transformers) (0.8)\n",
      "Requirement already satisfied, skipping upgrade: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch>=1.6.0->sentence-transformers) (3.7.4.3)\n",
      "Requirement already satisfied, skipping upgrade: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (0.0.43)\n",
      "Requirement already satisfied, skipping upgrade: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (2019.12.20)\n",
      "Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (2.23.0)\n",
      "Requirement already satisfied, skipping upgrade: tokenizers<0.11,>=0.10.1 in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (0.10.1)\n",
      "Requirement already satisfied, skipping upgrade: packaging in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (20.9)\n",
      "Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (3.4.0)\n",
      "Requirement already satisfied, skipping upgrade: filelock in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (3.0.12)\n",
      "Requirement already satisfied, skipping upgrade: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sentence-transformers) (1.0.0)\n",
      "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from nltk->sentence-transformers) (1.15.0)\n",
      "Requirement already satisfied, skipping upgrade: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers<5.0.0,>=3.1.0->sentence-transformers) (7.1.2)\n",
      "Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (3.0.4)\n",
      "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (1.24.3)\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (2020.12.5)\n",
      "Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (2.10)\n",
      "Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers<5.0.0,>=3.1.0->sentence-transformers) (2.4.7)\n",
      "Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers<5.0.0,>=3.1.0->sentence-transformers) (3.4.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install -U sentence-transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-2N6OpUEoJw5"
   },
   "source": [
    "# Selecting transformer and Cross Encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "d154TGcmoIjL"
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer, util, CrossEncoder\n",
    "import torch\n",
    "import time\n",
    "\n",
    "encoder = SentenceTransformer('msmarco-distilbert-base-v2')\n",
    "cross_encoder = CrossEncoder('cross-encoder/ms-marco-TinyBERT-L-6')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V5KbffTGfE9I"
   },
   "source": [
    "# Initializing corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l_L7TL9AfECp"
   },
   "outputs": [],
   "source": [
    "corpus = list(CORD19Dictionary.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FbDuU-ZVbp_z"
   },
   "source": [
    "# Creating the embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BcWILEtEpSWq"
   },
   "source": [
    "Encoding the papers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 100,
     "referenced_widgets": [
      "b4e5ac60951746dcbd8e263a6c1b3cbd",
      "2513c9df07e64820a11655a80505a1f0",
      "dfdca8daf8784a3f9895461ba8789fe3",
      "0ee152d36c8240f39ef86a0cbfbe1b64",
      "26aae27ca2154ef8929159efe55cbc7f",
      "3019714788cb417c81cf7f2ee30260eb",
      "5fb688f01e1d4d10970eb59d3ae8efba",
      "83071c19219e403f97bd7d6ae6a291e9"
     ]
    },
    "id": "cq6N8lvjr6d8",
    "outputId": "ce62d97e-3e38-41dd-ce31-2d5f1c7c1cee"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4e5ac60951746dcbd8e263a6c1b3cbd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Batches', max=6793.0, style=ProgressStyle(description_wid…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "CPU times: user 1min 48s, sys: 32.6 s, total: 2min 20s\n",
      "Wall time: 2min 16s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "corpus_embeddings = encoder.encode(corpus, convert_to_tensor=True, show_progress_bar=True,device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zx8ai0pkBX5a"
   },
   "source": [
    "# Saving corpus as tensors to drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "o8n4TfTGBU02"
   },
   "outputs": [],
   "source": [
    "corpus_embeddings_path = r\"/content/drive/My Drive/AI_4/corpus_embeddings_6000_CrossEncoder.pt\"\n",
    "torch.save(corpus_embeddings,corpus_embeddings_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rv0gd5jB-rW2"
   },
   "source": [
    "# Loading embeddings if have been created and saved\n",
    "\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KnuBqIvkqyjp"
   },
   "outputs": [],
   "source": [
    "corpus_embeddings_path = r\"/content/drive/My Drive/AI_4/corpus_embeddings_6000_CrossEncoder.pt\"\n",
    "with open(corpus_embeddings_path, 'rb') as f:\n",
    "    corpus_embeddings = torch.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OBnA24H--5AZ"
   },
   "source": [
    "# Evaluation\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VQfwoEZv7wzN"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from  nltk import tokenize\n",
    "from termcolor import colored\n",
    "\n",
    "\n",
    "def paperTitle(answer,SentenceMap):\n",
    "  record = SentenceMap[answer]\n",
    "  print(\"Paper title:\",record[1])\n",
    "  print(\"Paper id:   \",record[0])  \n",
    "  \n",
    "def evaluation(query_list,top_k,resultsDf):\n",
    "  query_answers = []\n",
    "  scores = []\n",
    "\n",
    "  for query in query_list:\n",
    "    #Encode the query using the bi-encoder and find potentially relevant corpus\n",
    "    start_time = time.time()\n",
    "    question_embedding = encoder.encode(query, convert_to_tensor=True,device='cuda')\n",
    "    hits = util.semantic_search(question_embedding, corpus_embeddings, top_k=top_k)\n",
    "    hits = hits[0]  # Get the hits for the first query\n",
    "\n",
    "    #Now, score all retrieved corpus with the cross_encoder\n",
    "    cross_inp = [[query, corpus[hit['corpus_id']]] for hit in hits]\n",
    "    cross_scores = cross_encoder.predict(cross_inp)\n",
    "  \n",
    "    #Sort results by the cross-encoder scores\n",
    "    for idx in range(len(cross_scores)):\n",
    "        hits[idx]['cross-score'] = cross_scores[idx]\n",
    "\n",
    "    hits = sorted(hits, key=lambda x: x['cross-score'], reverse=True)\n",
    "    end_time = time.time()\n",
    "\n",
    "    #Output of top-5 hits\n",
    "    print(\"\\n\\n======================\\n\\n\")\n",
    "    print(\"Query:\",colored(query,'green') )\n",
    "    \n",
    "    print(\"Results (after {:.3f} seconds):\".format(end_time - start_time))\n",
    "    iter=0\n",
    "    for hit in hits[0:top_k]:\n",
    "        print(\"\\n-> \",iter+1)\n",
    "        answer = ' '.join([re.sub(r\"^\\[.*\\]\", \"\", x) for x in corpus[hit['corpus_id']].split()])\n",
    "        if len(tokenize.word_tokenize(answer)) > 1:\n",
    "          print(\"Score: {:.4f}\".format(hit['cross-score']))\n",
    "          \n",
    "          paperTitle(corpus[hit['corpus_id']],CORD19Dictionary)\n",
    "          print(\"Anser size: \",len(tokenize.word_tokenize(answer)))\n",
    "          print(\"Anser: \")\n",
    "          if iter==0:\n",
    "            query_answers.append(answer)\n",
    "            scores.append(hit['cross-score'].item())\n",
    "          iter+=1\n",
    "          print(colored(answer,'yellow'))\n",
    "          \n",
    "  \n",
    "  resultsDf['Model_answer'] = query_answers\n",
    "  resultsDf['Cosine_similarity'] = scores\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "I_rAM46T8pjg",
    "outputId": "006ab534-7ced-412c-a03d-5dd2bc5c0c3b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat are the coronoviruses?\u001b[0m\n",
      "Results (after 0.839 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0639\n",
      "Paper title: Citation: Interactions Between Enteroviruses and the Inflammasome: New Insights Into Viral Pathogenesis\n",
      "Paper id:    423e1f15afb86012057acacc26d0766aa4bc582a\n",
      "Anser size:  7\n",
      "Anser: \n",
      "\u001b[33mEnteroviruses are the members of Picornaviridae.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0185\n",
      "Paper title: Full Genome Virus Detection in Fecal Samples Using Sensitive Nucleic Acid Preparation, Deep Sequencing, and a Novel Iterative Sequence Classification Algorithm\n",
      "Paper id:    ab98d1b125aa0704e63adef426b27abd32e935f0\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mCosavirus is a new genus in the Picornaviridae family first described in 2008   .\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0073\n",
      "Paper title: Identification of diverse viruses in upper respiratory samples in dromedary camels from United Arab Emirates\n",
      "Paper id:    04b5f15cca91a7b810216682780f8ea6e1ab3046\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mOrthonairoviruses.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat was discovered in Wuhuan in December 2019?\u001b[0m\n",
      "Results (after 0.525 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.7336\n",
      "Paper title: Transmission routes of 2019-nCoV and controls in dental practice\n",
      "Paper id:    9756bb3c608ed790d2306fc8db815a694eeca45f\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mAn emergent pneumonia outbreak originated in Wuhan City, in the late December 2019 1 .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0006\n",
      "Paper title: Molecular Sciences Effects of AntagomiRs on Different Lung Diseases in Human, Cellular, and Animal Models\n",
      "Paper id:    3aed588044335032787a5eb91ee61afadcd4a006\n",
      "Anser size:  6\n",
      "Anser: \n",
      "\u001b[33m2019  or Liu et al.,\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0002\n",
      "Paper title: Estimated effectiveness of symptom and risk screening to prevent the spread of COVID-19\n",
      "Paper id:    a70e7c4d8ee484ce956e91c8700d0c9310bbdbbc\n",
      "Anser size:  6\n",
      "Anser: \n",
      "\u001b[33m2020; Liu et al.,\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is Coronovirus Disease 2019?\u001b[0m\n",
      "Results (after 0.523 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.6516\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.2219\n",
      "Paper title: \n",
      "Paper id:    19ff77e874c0706f794908e9b6878314671d385a\n",
      "Anser size:  9\n",
      "Anser: \n",
      "\u001b[33mNaming 2019-nCoV as SARS-CoV-2 is therefore truly misleading.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.1593\n",
      "Paper title: \n",
      "Paper id:    82210c1cb5ac59acd1468cedcaf6fb8d951f4903\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mThe infective pathogen was later identified as a novel coronavirus, called 2019-nCoV  .\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is COVID-19?\u001b[0m\n",
      "Results (after 0.551 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.9631\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.6902\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.1879\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is caused by SARS-COV2?\u001b[0m\n",
      "Results (after 0.551 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.3216\n",
      "Paper title: \n",
      "Paper id:    0eb44c0cc59184754a0a2cd8ee3c8b2302a8927c\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mWe thus assumed that a SARS-related CoV is involved in the outbreak.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0962\n",
      "Paper title: Middle East respiratory syndrome coronavirus infection: virus-host cell interactions and implications on pathogenesis\n",
      "Paper id:    12f712c348c26e092759d804778defe2d2d4af6f\n",
      "Anser size:  7\n",
      "Anser: \n",
      "\u001b[33mThe SARS-CoV can infect human macrophages.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0673\n",
      "Paper title: Potential Factors Influencing Repeated SARS Outbreaks in China\n",
      "Paper id:    655537fc8cc52bccf43cf7189ab060d3097caa7a\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mThe risk of SARS-CoV-2 infection will remain for a long time.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow is COVID-19 spread?\u001b[0m\n",
      "Results (after 0.547 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.9799\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mThe COVID-19 has then rapidly spread to all over China and the world.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.9631\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  18\n",
      "Anser: \n",
      "\u001b[33mIt is found that the COVID-19 can be transmitted through droplets, contact, aerosol, etc.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.2594\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhere was COVID-19 discovered?\u001b[0m\n",
      "Results (after 0.530 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.9490\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  17\n",
      "Anser: \n",
      "\u001b[33mThe epidemic of COVID-19 is caused by a novel virus first detected in Wuhan, China.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.4963\n",
      "Paper title: The novel coronavirus outbreak in Wuhan, China\n",
      "Paper id:    5ba8056230c17ec133169d79aacf61ed7d4b458b\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mThe COVID-19 has then rapidly spread to all over China and the world.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0428\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow does coronavirus spread?\u001b[0m\n",
      "Results (after 0.548 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.1335\n",
      "Paper title: Analysis of the codon usage pattern in Middle East Respiratory Syndrome Coronavirus\n",
      "Paper id:    627ada5c21fb8d0e43b37999fa66bf41ca36c353\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mThis may hint that coronavirus does not spread so widely in humans.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0041\n",
      "Paper title: Tropical Medicine and Infectious Disease Potential Intermediate Hosts for Coronavirus Transmission: No Evidence of Clade 2c Coronaviruses in Domestic Livestock from Ghana\n",
      "Paper id:    95cc317541d97e3dbaa1662894fdbed842098910\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mcoronavirus.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0041\n",
      "Paper title: Population genetics, community of parasites, and resistance to rodenticides in an urban brown rat (Rattus norvegicus) population\n",
      "Paper id:    c8d60caf44017989b3b9633350fc1d2efda570a5\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mCoronavirus.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "top_k = 3\n",
    "evaluation(query_list,top_k,queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F36KUgWNId-0",
    "outputId": "31de041f-ee87-426f-f61c-ce66ebdda295"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow long can the coronavirus survive on surfaces?\u001b[0m\n",
      "Results (after 0.537 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.9850\n",
      "Paper title: Outbreak of Novel Coronavirus (SARS-Cov-2): First Evidences From International Scientific Literature and Pending Questions\n",
      "Paper id:    7b7c71218f8d7ea1a1f8f702e4262b839bf7cc8a\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mOn inanimate surfaces, human coronaviruses can remain infectious for up to 9 days.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.7655\n",
      "Paper title: Characterisation of the canine faecal virome in healthy dogs and dogs with acute diarrhoea using shotgun metagenomics\n",
      "Paper id:    fcb1ba715b2516823fee057cbb0f8276c76d19d7\n",
      "Anser size:  21\n",
      "Anser: \n",
      "\u001b[33mCanine coronavirus can be shed in faeces in high numbers for up to 156 days [44, 45] .\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.1069\n",
      "Paper title: Human Coronaviruses: Insights into Environmental Resistance and Its Influence on the Development of New Antiseptic Strategies\n",
      "Paper id:    d171f82b892a2afafc2bc8a5458219dc04c8fd8d\n",
      "Anser size:  21\n",
      "Anser: \n",
      "\u001b[33mHuman coronavirus infections occur mainly in winter, with a short incubation time [19, 23, 24] .\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat means COVID-19?\u001b[0m\n",
      "Results (after 0.534 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.9272\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.7040\n",
      "Paper title: \n",
      "Paper id:    19ff77e874c0706f794908e9b6878314671d385a\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mThe new name is also not consistent with the disease name COVID-19.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.5282\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mIs COVID19 worse than flue?\u001b[0m\n",
      "Results (after 0.521 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0644\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  11\n",
      "Anser: \n",
      "\u001b[33mIn comparison, COVID-19 showed similar trends with SARS patients  .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0359\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0022\n",
      "Paper title: The effect of corticosteroids on mortality of patients with influenza pneumonia: a systematic review and meta-analysis\n",
      "Paper id:    ff220214e91fabc8d302d1605cd9bac44fac507f\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mCorticosteroids could increase mortality in patients with influenza pneumonia.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhen the vaccine will be ready?\u001b[0m\n",
      "Results (after 0.536 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.2100\n",
      "Paper title: Mast cells and influenza A virus: association with allergic responses and beyond\n",
      "Paper id:    29621887690af716dac0c244eeb95bce74fa8755\n",
      "Anser size:  10\n",
      "Anser: \n",
      "\u001b[33mCurrent vaccine strategies take approximately 6 months for production.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.1330\n",
      "Paper title: Rapid and simple colorimetric detection of multiple influenza viruses infecting humans using a reverse transcriptional loop- mediated isothermal amplification (RT-LAMP) diagnostic platform\n",
      "Paper id:    dd12c39ca963dca8336d7f30c8842d892ec8236c\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mHowever, vaccine production usually takes 6-12 months to prepare for newly emerging viruses.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.1195\n",
      "Paper title: Vaccination to Conserved Influenza Antigens in Mice Using a Novel Simian Adenovirus Vector, PanAd3, Derived from the Bonobo Pan paniscus\n",
      "Paper id:    532e417a66dbe4822a3f8f9b496c105ccc7dd412\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mNew vaccines are often required, and take about 6 months to become available  .\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhats the proteins that consist COVID-19?\u001b[0m\n",
      "Results (after 0.532 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0004\n",
      "Paper title: Integrin b3 Is Required in Infection and Proliferation of Classical Swine Fever Virus\n",
      "Paper id:    e50473adb66bac4a176d80051d63f415d2dbd5a8\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mCSFV contains 4 structural proteins: C, Erns, E1 and E2.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0004\n",
      "Paper title: Proteome and phosphoproteome analysis of honeybee (Apis mellifera) venom collected from electrical stimulation and manual extraction of the venom gland Proteome and phosphoproteome analysis of honeybee (Apis mellifera) venom collected from electrical stimulation and manual extraction of the venom gland\n",
      "Paper id:    1c8a0fb2f60c243f71d16d5fefb5b51d7978869e\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mIn GV, 27 proteins were specifically expressed: 4 toxins and 23 non-toxins.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0002\n",
      "Paper title: The Interplay between Dengue Virus and the Human Innate Immune System: A Game of Hide and Seek\n",
      "Paper id:    ac8b5e9b4a49a1062eddf4fc48a19778e66e9a78\n",
      "Anser size:  3\n",
      "Anser: \n",
      "\u001b[33mThis proteins.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhats the symptoms of COVID-19?\u001b[0m\n",
      "Results (after 0.525 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.1345\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0006\n",
      "Paper title: Characterization of Host and Bacterial Contributions to Lung Barrier Dysfunction Following Co-infection with 2009 Pandemic Influenza and Methicillin Resistant Staphylococcus aureus\n",
      "Paper id:    edee1fd45587a0a71d88a5db58cc81342840e2f6\n",
      "Anser size:  5\n",
      "Anser: \n",
      "\u001b[33mInitial signs and symptoms include\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0002\n",
      "Paper title: Effect of Pullet Vaccination on Development and Longevity of Immunity\n",
      "Paper id:    e192e65a6546583fe49086c4d3ac29a0620d5bd5\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mClinical Signs\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow can I prevent COVID-19?\u001b[0m\n",
      "Results (after 0.531 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.2469\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.1603\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0029\n",
      "Paper title: Identify-Isolate-Inform: A Modified Tool for Initial Detection and Management of Middle East Respiratory Syndrome Patients in the Emergency Department\n",
      "Paper id:    e8ae9d6178f8322e2f9b2453ef13bb312427bd15\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mPrevention of MERS-CoV transmission involves avoiding exposure.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat treatments are available for COVID-19?\u001b[0m\n",
      "Results (after 0.531 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.9569\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  17\n",
      "Anser: \n",
      "\u001b[33mAs effective drugs for SARS, hormones and interferons can also be used to treat COVID-19  .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0802\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0433\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mIs hand sanitizer effective against COVID-19?\u001b[0m\n",
      "Results (after 0.534 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.2058\n",
      "Paper title: Respiratory viral infections in children with asthma: do they matter and can we prevent them?\n",
      "Paper id:    bbc2824ce7dff3d23d060b7abbe96cba28095fb8\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mThe use of alcohol-based hand sanitizers is also effective [54, 55] .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0120\n",
      "Paper title: Cell Discovery Phase-adjusted estimation of the number of Coronavirus Disease 2019 cases in Wuhan, China\n",
      "Paper id:    6abb30ae61aa5e41f16a28b9437940d5d76d745b\n",
      "Anser size:  19\n",
      "Anser: \n",
      "\u001b[33mIn response to the outbreak of COVID-19, a series of prompt public health measures have been taken.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0036\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mAm I at risk for serious complications from COVID-19 if I smoke cigarettes?\u001b[0m\n",
      "Results (after 0.523 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0151\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mreported that people who have not been exposed to SARS-CoV-2 are all susceptible to COVID-19  .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0042\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0014\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mAre there any FDA-approved drugs (medicines) for COVID-19?\u001b[0m\n",
      "Results (after 0.519 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0156\n",
      "Paper title: \n",
      "Paper id:    19ff77e874c0706f794908e9b6878314671d385a\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mThe new name is also not consistent with the disease name COVID-19.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0143\n",
      "Paper title: Clinical Medicine Incubation Period and Other Epidemiological Characteristics of 2019 Novel Coronavirus Infections with Right Truncation: A Statistical Analysis of Publicly Available Case Data\n",
      "Paper id:    210a892deb1c61577f6fba58505fd65356ce6636\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mIt remains to be seen if this will be the case for COVID-19 as well.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0028\n",
      "Paper title: Human Ebola virus infection in West Africa: a review of available therapeutic agents that target different steps of the life cycle of Ebola virus-mutable host cell therapeutic targets for Ebola virus, Cocktail therapeutic intervention for RNA virus Multilingual abstract\n",
      "Paper id:    06190bfcbc53a5d5d17e0a60a3a0f6488d8ae1db\n",
      "Anser size:  11\n",
      "Anser: \n",
      "\u001b[33mThese medications are FDA-approved for the treatment of other diseases.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow are people tested?\u001b[0m\n",
      "Results (after 0.520 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0092\n",
      "Paper title: A Human DPP4-Knockin Mouse's Susceptibility to Infection by Authentic and Pseudotyped MERS-CoV\n",
      "Paper id:    874e540a730ee1060365af8d2caa03f537508e33\n",
      "Anser size:  11\n",
      "Anser: \n",
      "\u001b[33mStudent's t-tests were used to assess differences between groups.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0075\n",
      "Paper title: Virology Journal A focus reduction neutralization assay for hepatitis C virus neutralizing antibodies\n",
      "Paper id:    ee8dca216514deeed4c9415bc2ad8a78dc3d9670\n",
      "Anser size:  11\n",
      "Anser: \n",
      "\u001b[33mStudent's t-test was used to compare data between groups.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0006\n",
      "Paper title: Ecohealth research in Southeast Asia: past, present and the way forward\n",
      "Paper id:    1495c1fa93db3b9a5d12b5ae15ff0c8639b83452\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mHow can these be tested in practice?\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhy is the disease being called coronavirus disease 2019, COVID-19?\u001b[0m\n",
      "Results (after 0.528 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.9373\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.9238\n",
      "Paper title: Cell Discovery Phase-adjusted estimation of the number of Coronavirus Disease 2019 cases in Wuhan, China\n",
      "Paper id:    6abb30ae61aa5e41f16a28b9437940d5d76d745b\n",
      "Anser size:  20\n",
      "Anser: \n",
      "\u001b[33mWorld Health Organization (WHO) now has named the disease Coronavirus Disease 2019 (COVID- 19) 3 .\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.8515\n",
      "Paper title: \n",
      "Paper id:    82210c1cb5ac59acd1468cedcaf6fb8d951f4903\n",
      "Anser size:  14\n",
      "Anser: \n",
      "\u001b[33mThe infective pathogen was later identified as a novel coronavirus, called 2019-nCoV  .\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mAm I at risk for COVID-19 from mail, packages, or products?\u001b[0m\n",
      "Results (after 0.520 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.1850\n",
      "Paper title: First two months of the 2019 Coronavirus Disease (COVID-19) epidemic in China: real- time surveillance and evaluation with a second derivative model\n",
      "Paper id:    469ed0f00c09e2637351c9735c306f27acf3aace\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mThis is particularly true for the COVID-19.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0283\n",
      "Paper title: Systematic Comparison of Two Animal-to-Human Transmitted Human Coronaviruses: SARS-CoV-2 and SARS-CoV\n",
      "Paper id:    f294f0df7468a8ac9e27776cc15fa20297a9f040\n",
      "Anser size:  16\n",
      "Anser: \n",
      "\u001b[33mreported that people who have not been exposed to SARS-CoV-2 are all susceptible to COVID-19  .\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0033\n",
      "Paper title: \n",
      "Paper id:    af000c5a8e181550fd16291e5d4f0f70ca9161a1\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mCOVID-19: coronavirus disease 2019; PPE: personal protective equipment.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is community spread?\u001b[0m\n",
      "Results (after 0.526 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.2350\n",
      "Paper title: An Opportunistic Pathogen Afforded Ample Opportunities: Middle East Respiratory Syndrome Coronavirus\n",
      "Paper id:    32da24606ad160166f08cf05349eaadd580ccff0\n",
      "Anser size:  9\n",
      "Anser: \n",
      "\u001b[33mCommunity spread and subclinical transmission need more attention.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0135\n",
      "Paper title: People at Risk of Influenza Pandemics: The Evolution of Perception and Behavior\n",
      "Paper id:    51f8792fd26cd2c094c1b2d0e5539902fb6221da\n",
      "Anser size:  15\n",
      "Anser: \n",
      "\u001b[33mEfforts were then put into preventing spread of the disease at the community level.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0002\n",
      "Paper title: Comparative Analysis of the Effectiveness of Three Immunization Strategies in Controlling Disease Outbreaks in Realistic Social Networks\n",
      "Paper id:    b16b23aad25d88c3af9ccd50b754cd4d9e8762fe\n",
      "Anser size:  3\n",
      "Anser: \n",
      "\u001b[33mCommunity-Bridge Immunization.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mHow can I protect myself?\u001b[0m\n",
      "Results (after 0.529 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0502\n",
      "Paper title: BMC Public Health Healthcare workers' attitudes to working during pandemic influenza: a qualitative study\n",
      "Paper id:    c337fa83ebb25e4600c0f9333ee0cb0fa938e947\n",
      "Anser size:  8\n",
      "Anser: \n",
      "\u001b[33mGet as much protection as you can.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0354\n",
      "Paper title: Need of surveillance response systems to combat Ebola outbreaks and other emerging infectious diseases in African countries\n",
      "Paper id:    70f3c90a651224f9292378da905af4ec635d5f43\n",
      "Anser size:  18\n",
      "Anser: \n",
      "\u001b[33mMoreover, people who don't have the knowledge should be educated on how to protect themselves.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0019\n",
      "Paper title: a Stakeholder Survey on live Bird market closures policy for controlling Highly pathogenic avian influenza in Vietnam\n",
      "Paper id:    17fe16cf66ebbe693a2e75dda11d14513fec7519\n",
      "Anser size:  13\n",
      "Anser: \n",
      "\u001b[33mTo mitigate these risks, the following safeguards were put in place.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWhat is a novel coronavirus?\u001b[0m\n",
      "Results (after 0.520 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.1118\n",
      "Paper title: Population genetics, community of parasites, and resistance to rodenticides in an urban brown rat (Rattus norvegicus) population\n",
      "Paper id:    c8d60caf44017989b3b9633350fc1d2efda570a5\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mCoronavirus.\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.1118\n",
      "Paper title: Tropical Medicine and Infectious Disease Potential Intermediate Hosts for Coronavirus Transmission: No Evidence of Clade 2c Coronaviruses in Domestic Livestock from Ghana\n",
      "Paper id:    95cc317541d97e3dbaa1662894fdbed842098910\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mcoronavirus.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0265\n",
      "Paper title: Retargeting of Viruses to Generate Oncolytic Agents\n",
      "Paper id:    bd44d72a9c41b1c382bd180da10a1f7ef38d2d56\n",
      "Anser size:  2\n",
      "Anser: \n",
      "\u001b[33mCoronaviruses.\u001b[0m\n",
      "\n",
      "\n",
      "======================\n",
      "\n",
      "\n",
      "Query: \u001b[32mWas Harry Potter a good magician?\u001b[0m\n",
      "Results (after 0.515 seconds):\n",
      "\n",
      "->  1\n",
      "Score: 0.0002\n",
      "Paper title: Immunoproteomic analysis of bacterial proteins of Actinobacillus pleuropneumoniae serotype 1\n",
      "Paper id:    47fb645312a069e65bb7557c58204244c3c92953\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mThe vaccines elicited humoral immune responses and protective efficacy in mice  .\u001b[0m\n",
      "\n",
      "->  2\n",
      "Score: 0.0002\n",
      "Paper title: \n",
      "Paper id:    fa137f1562d599f03605b83bc68f91e5105110d9\n",
      "Anser size:  5\n",
      "Anser: \n",
      "\u001b[33mEhrlich's magic bullet.\u001b[0m\n",
      "\n",
      "->  3\n",
      "Score: 0.0002\n",
      "Paper title: Knowledge and attitudes of university students toward pandemic influenza: a cross-sectional study from Turkey\n",
      "Paper id:    545def8771357b4cb2875f5795a0760e97534cc9\n",
      "Anser size:  12\n",
      "Anser: \n",
      "\u001b[33mSurprisingly, a higher proportion believed that herbal remedies were effective.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "top_k = 3\n",
    "evaluation(myquery_list,top_k,myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zd29996wfURy"
   },
   "source": [
    "# Overall results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w5o0GT6OglvS"
   },
   "source": [
    "## 6000 papers with no summarization \n",
    "---\n",
    "\n",
    "### Time needed for creating the embeddings: \n",
    "- CPU times: \n",
    "  - user 13min 10s\n",
    "  - sys: 5min 40s\n",
    "  - total: 18min 51s\n",
    "- Wall time: 18min 26s\n",
    "\n",
    "### Remarks\n",
    "Best results among the notebooks so far, almost 5/7 questions are answered and from mine 7/17. I expected better results since Cross Encoders enhance much the performance of Sentence Bert.\n",
    "\n",
    "__Top-k__ \n",
    "\n",
    "Top-2 and 3 have lots of answers, as I noticed that are better that the first one. Also good results and with some tunning would be nearly to the wanted.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tYoPMvbLlWr0"
   },
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 518
    },
    "id": "HZa1K9iDtMw_",
    "outputId": "fb7f4afe-6345-4a65-b652-9bef9602f9ef"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes.</td>\n",
       "      <td>C oronaviruses (CoV) represent a diverse family of positivesense RNA viruses capable of causing respiratory and enteric disease in human and animal hosts.</td>\n",
       "      <td>0.099706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>A mysterious illness causing pneumonia in December 2019 in Wuhan, China is now growing into a potential pandemic.</td>\n",
       "      <td>0.919291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.</td>\n",
       "      <td>CDC considered the 2019-nCoV as a possible pathogen causing the outbreak  .</td>\n",
       "      <td>0.279708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.</td>\n",
       "      <td>For COVID-19, there is so far limited evidence for specific risk factors; we therefore assumed that at most 40% of travellers would be aware of a potential exposure.</td>\n",
       "      <td>0.827424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.</td>\n",
       "      <td>The SARS outbreak of 2002-2003 was caused by SARS-CoV, a novel coronavirus.</td>\n",
       "      <td>0.951182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.</td>\n",
       "      <td>At the same time, there is great concern about potential public health consequences if COVID-19 spreads to developing countries that lack health infrastructure and resources to combat it effectively (de Salazar et al., 2020) .</td>\n",
       "      <td>0.971672</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>Three cases of COVID-19 were confirmed on 24 January, the first cases in Europe.</td>\n",
       "      <td>0.938341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.</td>\n",
       "      <td>Corona viruses are transmitted via airborne zoonotic droplets, and viral replication occurs in the ciliated epithelium, resulting in cellular damage and inflammatory reactions at the site of infection [3, 4] .</td>\n",
       "      <td>0.893843</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...          0.099706\n",
       "1  What was discovered in Wuhuan in December 2019?  ...          0.919291\n",
       "2                What is Coronovirus Disease 2019?  ...          0.279708\n",
       "3                                What is COVID-19?  ...          0.827424\n",
       "4                     What is caused by SARS-COV2?  ...          0.951182\n",
       "5                          How is COVID-19 spread?  ...          0.971672\n",
       "6                   Where was COVID-19 discovered?  ...          0.938341\n",
       "7                     How does coronavirus spread?  ...          0.893843\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 811
    },
    "id": "7vxQKoC3tMxC",
    "outputId": "6c61de15-b790-4960-a438-deb14d72fda7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How long can the coronavirus survive on surfaces?</td>\n",
       "      <td>Despite the general view that enveloped viruses are fragile outside the host, several coronaviruses remain infective after drying on surfaces for more than 24 h as reviewed by Otter et al.</td>\n",
       "      <td>0.957455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What means COVID-19?</td>\n",
       "      <td>By submitting their studies to \"COVID-19 Open,\" researchers can share their data while meeting their need to retain authorship, document precedence and facilitate international scientific cooperation in the response to this emergency.</td>\n",
       "      <td>0.399066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Is COVID19 worse than flue?</td>\n",
       "      <td>In all types of influenza virus, mortality was higher in those treated with corticosteroids than in controls, although symptoms were more rapidly progressive patients and the risk of ARDS higher in patients infected with H7N9 [1, 2] .</td>\n",
       "      <td>0.008936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>When the vaccine will be ready?</td>\n",
       "      <td>If an FDA-approved live attenuated vaccine is required, it is recommended that it be given at least 4 weeks before the first or 4 weeks after the third study injection.</td>\n",
       "      <td>0.410219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Whats the proteins that consist COVID-19?</td>\n",
       "      <td>Four structural proteins are encoded by the CoV genomes: spike (S), membrane (M), envelope (E), and nucleocapsid (N).</td>\n",
       "      <td>0.020674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Whats the symptoms of COVID-19?</td>\n",
       "      <td>The novel pneumonia was named as Corona Virus by World Health Organization (WHO) ( the common symptoms of COVID-19 at illness onset were fever, fatigue, dry cough, myalgia, and dyspnea 3 .</td>\n",
       "      <td>0.939072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>How can I prevent COVID-19?</td>\n",
       "      <td>For COVID-19, there is so far limited evidence for specific risk factors; we therefore assumed that at most 40% of travellers would be aware of a potential exposure.</td>\n",
       "      <td>0.366725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What treatments are available for COVID-19?</td>\n",
       "      <td>Although treatment with neuraminidase inhibitors (oseltamivir or zanamivir) is recommended in all patients with suspected or confirmed influenza requiring hospitalization  their use in non-severe influenza could be more harmful than beneficial because of the possibility of selection of resistant mutants  .</td>\n",
       "      <td>0.001212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Is hand sanitizer effective against COVID-19?</td>\n",
       "      <td>At the same time, there is great concern about potential public health consequences if COVID-19 spreads to developing countries that lack health infrastructure and resources to combat it effectively (de Salazar et al., 2020) .</td>\n",
       "      <td>0.014746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Am I at risk for serious complications from COVID-19 if I smoke cigarettes?</td>\n",
       "      <td>For COVID-19, there is so far limited evidence for specific risk factors; we therefore assumed that at most 40% of travellers would be aware of a potential exposure.</td>\n",
       "      <td>0.164191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Are there any FDA-approved drugs (medicines) for COVID-19?</td>\n",
       "      <td>No identified contact of the three cases has been confirmed with COVID-19.</td>\n",
       "      <td>0.016031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>How are people tested?</td>\n",
       "      <td>were tested.</td>\n",
       "      <td>0.086004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Why is the disease being called coronavirus disease 2019, COVID-19?</td>\n",
       "      <td>As of February 20, 2020, the 2019 novel coronavirus (now named SARS-CoV-2, causing the disease COVID-19) has caused over 75,000 confirmed cases inside of China and has spread to 25 other countries (World Health Organization, 2020b) .</td>\n",
       "      <td>0.970304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Am I at risk for COVID-19 from mail, packages, or products?</td>\n",
       "      <td>For COVID-19, there is so far limited evidence for specific risk factors; we therefore assumed that at most 40% of travellers would be aware of a potential exposure.</td>\n",
       "      <td>0.718535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>What is community spread?</td>\n",
       "      <td>Community spread and subclinical transmission need more attention.</td>\n",
       "      <td>0.235021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>How can I protect myself?</td>\n",
       "      <td>Without this, they may not know how to take the necessary steps to protect themselves.</td>\n",
       "      <td>0.053354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>What is a novel coronavirus?</td>\n",
       "      <td>Coronavirus.</td>\n",
       "      <td>0.111760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Was Harry Potter a good magician?</td>\n",
       "      <td>Protective Efficacy in TLR3 / TLR7 / Mice To further investigate the contribution of innate immunity to early protection, transgenic mice deficient in both TLR3 and TLR7 were employed (Diebold et al., 2004; Guillot et al., 2005) .</td>\n",
       "      <td>0.000176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          Query  ... Cosine_similarity\n",
       "0                             How long can the coronavirus survive on surfaces?  ...          0.957455\n",
       "1                                                          What means COVID-19?  ...          0.399066\n",
       "2                                                   Is COVID19 worse than flue?  ...          0.008936\n",
       "3                                               When the vaccine will be ready?  ...          0.410219\n",
       "4                                     Whats the proteins that consist COVID-19?  ...          0.020674\n",
       "5                                               Whats the symptoms of COVID-19?  ...          0.939072\n",
       "6                                                   How can I prevent COVID-19?  ...          0.366725\n",
       "7                                   What treatments are available for COVID-19?  ...          0.001212\n",
       "8                                 Is hand sanitizer effective against COVID-19?  ...          0.014746\n",
       "9   Am I at risk for serious complications from COVID-19 if I smoke cigarettes?  ...          0.164191\n",
       "10                   Are there any FDA-approved drugs (medicines) for COVID-19?  ...          0.016031\n",
       "11                                                       How are people tested?  ...          0.086004\n",
       "12          Why is the disease being called coronavirus disease 2019, COVID-19?  ...          0.970304\n",
       "13                  Am I at risk for COVID-19 from mail, packages, or products?  ...          0.718535\n",
       "14                                                    What is community spread?  ...          0.235021\n",
       "15                                                    How can I protect myself?  ...          0.053354\n",
       "16                                                 What is a novel coronavirus?  ...          0.111760\n",
       "17                                            Was Harry Potter a good magician?  ...          0.000176\n",
       "\n",
       "[18 rows x 3 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kgPN-tT4tODl"
   },
   "source": [
    "## 9000 papers with no summarization \n",
    "---\n",
    "\n",
    "Session crashed due to RAM\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bFwQUzLvtOq7"
   },
   "source": [
    "## 6000 papers with paraphrase-distilroberta-base-v1 model and summarization \n",
    "---\n",
    "\n",
    "### Time needed for creating the embeddings: \n",
    "- CPU times: \n",
    "  - user: 1min 18s\n",
    "  - sys: 22.8 s\n",
    "  - total: 1min 37s\n",
    "- Wall time: 1min 37s\n",
    "\n",
    "### Remarks\n",
    "Not good results. From these results I think that the BERT summarizer parameters were not the appropriate and I should experiment with them. I shouldn't have so strict summarization and I may over summarized the papers.\n",
    "\n",
    "\n",
    "__Top-k__ \n",
    "\n",
    "Not good.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cMCzJslstOq8"
   },
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 433
    },
    "id": "eRlKZFuctOq8",
    "outputId": "7852aa72-942b-4bf6-de07-a2b58d0dc8f3"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes.</td>\n",
       "      <td>CoV is a group of viruses that belong to the Coronaviridae family and the Nidovirales order.</td>\n",
       "      <td>0.083080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>2016 (Wu et al., ,</td>\n",
       "      <td>0.002247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.651608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.963140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.</td>\n",
       "      <td>We thus assumed that a SARS-related CoV is involved in the outbreak.</td>\n",
       "      <td>0.321600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.042323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>Strengthened surveillance of COVID-19 cases was implemented in France on 10 January 2020.</td>\n",
       "      <td>0.346553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.</td>\n",
       "      <td>This may hint that coronavirus does not spread so widely in humans.</td>\n",
       "      <td>0.133488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...          0.083080\n",
       "1  What was discovered in Wuhuan in December 2019?  ...          0.002247\n",
       "2                What is Coronovirus Disease 2019?  ...          0.651608\n",
       "3                                What is COVID-19?  ...          0.963140\n",
       "4                     What is caused by SARS-COV2?  ...          0.321600\n",
       "5                          How is COVID-19 spread?  ...          0.042323\n",
       "6                   Where was COVID-19 discovered?  ...          0.346553\n",
       "7                     How does coronavirus spread?  ...          0.133488\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 607
    },
    "id": "Xb1cb5I0tOq9",
    "outputId": "61aa0c3c-24db-430a-f4f5-89dee31c05ee"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How long can the coronavirus survive on surfaces?</td>\n",
       "      <td>Canine coronavirus can be shed in faeces in high numbers for up to 156 days [44, 45] .</td>\n",
       "      <td>0.765538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What means COVID-19?</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.927158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Is COVID19 worse than flue?</td>\n",
       "      <td>Corticosteroids could increase mortality in patients with influenza pneumonia.</td>\n",
       "      <td>0.002202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>When the vaccine will be ready?</td>\n",
       "      <td>Current vaccine strategies take approximately 6 months for production.</td>\n",
       "      <td>0.210023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Whats the proteins that consist COVID-19?</td>\n",
       "      <td>Of these proteins, 34 were shared across all strains ( Fig.</td>\n",
       "      <td>0.000441</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Whats the symptoms of COVID-19?</td>\n",
       "      <td>Initial signs and symptoms include</td>\n",
       "      <td>0.000551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>How can I prevent COVID-19?</td>\n",
       "      <td>Prevention of MERS-CoV transmission involves avoiding exposure.</td>\n",
       "      <td>0.002867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What treatments are available for COVID-19?</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.081656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Is hand sanitizer effective against COVID-19?</td>\n",
       "      <td>the effectiveness of handwashing and alcohol-based hand sanitizers.</td>\n",
       "      <td>0.065614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Am I at risk for serious complications from COVID-19 if I smoke cigarettes?</td>\n",
       "      <td>Tobacco/cigarette smoke exposure and influenza A virus infection.</td>\n",
       "      <td>0.001866</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Are there any FDA-approved drugs (medicines) for COVID-19?</td>\n",
       "      <td>There are currently no FDA-approved vaccines or therapeutics for public use.</td>\n",
       "      <td>0.005908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>How are people tested?</td>\n",
       "      <td>Student's t-tests were used to assess differences between groups.</td>\n",
       "      <td>0.009154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Why is the disease being called coronavirus disease 2019, COVID-19?</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.937286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Am I at risk for COVID-19 from mail, packages, or products?</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.003261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>What is community spread?</td>\n",
       "      <td>Community spread and subclinical transmission need more attention.</td>\n",
       "      <td>0.235021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>How can I protect myself?</td>\n",
       "      <td>Moreover, people who don't have the knowledge should be educated on how to protect themselves.</td>\n",
       "      <td>0.035414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>What is a novel coronavirus?</td>\n",
       "      <td>Coronavirus.</td>\n",
       "      <td>0.111760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Was Harry Potter a good magician?</td>\n",
       "      <td>It protected mice from a lethal ricin challenge without showing toxicity to the animals.</td>\n",
       "      <td>0.000182</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          Query  ... Cosine_similarity\n",
       "0                             How long can the coronavirus survive on surfaces?  ...          0.765538\n",
       "1                                                          What means COVID-19?  ...          0.927158\n",
       "2                                                   Is COVID19 worse than flue?  ...          0.002202\n",
       "3                                               When the vaccine will be ready?  ...          0.210023\n",
       "4                                     Whats the proteins that consist COVID-19?  ...          0.000441\n",
       "5                                               Whats the symptoms of COVID-19?  ...          0.000551\n",
       "6                                                   How can I prevent COVID-19?  ...          0.002867\n",
       "7                                   What treatments are available for COVID-19?  ...          0.081656\n",
       "8                                 Is hand sanitizer effective against COVID-19?  ...          0.065614\n",
       "9   Am I at risk for serious complications from COVID-19 if I smoke cigarettes?  ...          0.001866\n",
       "10                   Are there any FDA-approved drugs (medicines) for COVID-19?  ...          0.005908\n",
       "11                                                       How are people tested?  ...          0.009154\n",
       "12          Why is the disease being called coronavirus disease 2019, COVID-19?  ...          0.937286\n",
       "13                  Am I at risk for COVID-19 from mail, packages, or products?  ...          0.003261\n",
       "14                                                    What is community spread?  ...          0.235021\n",
       "15                                                    How can I protect myself?  ...          0.035414\n",
       "16                                                 What is a novel coronavirus?  ...          0.111760\n",
       "17                                            Was Harry Potter a good magician?  ...          0.000182\n",
       "\n",
       "[18 rows x 3 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wkDPatlQtPMr"
   },
   "source": [
    "## 9000 papers with summarization \n",
    "---\n",
    "\n",
    "### Time needed for creating the embeddings: \n",
    "- CPU times: \n",
    "  - user: 1min 48s\n",
    "  - sys: 32.6 s\n",
    "  - total: 2min 20s\n",
    "- Wall time: 2min 16s\n",
    "\n",
    "### Remarks\n",
    "Again not good results and this is due my summarization tunning.\n",
    "\n",
    "** Again I didn't have the time to re run and process again.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kv0_v66atPMu"
   },
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 433
    },
    "id": "5gP4Ati-tPMv",
    "outputId": "5f353704-e517-43bc-eda6-2dc4779ab62b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes.</td>\n",
       "      <td>Enteroviruses are the members of Picornaviridae.</td>\n",
       "      <td>0.063864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>An emergent pneumonia outbreak originated in Wuhan City, in the late December 2019 1 .</td>\n",
       "      <td>0.733632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.651608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.963140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.</td>\n",
       "      <td>We thus assumed that a SARS-related CoV is involved in the outbreak.</td>\n",
       "      <td>0.321600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.</td>\n",
       "      <td>The COVID-19 has then rapidly spread to all over China and the world.</td>\n",
       "      <td>0.979950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.</td>\n",
       "      <td>The epidemic of COVID-19 is caused by a novel virus first detected in Wuhan, China.</td>\n",
       "      <td>0.948981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.</td>\n",
       "      <td>This may hint that coronavirus does not spread so widely in humans.</td>\n",
       "      <td>0.133488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...          0.063864\n",
       "1  What was discovered in Wuhuan in December 2019?  ...          0.733632\n",
       "2                What is Coronovirus Disease 2019?  ...          0.651608\n",
       "3                                What is COVID-19?  ...          0.963140\n",
       "4                     What is caused by SARS-COV2?  ...          0.321600\n",
       "5                          How is COVID-19 spread?  ...          0.979950\n",
       "6                   Where was COVID-19 discovered?  ...          0.948981\n",
       "7                     How does coronavirus spread?  ...          0.133488\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 607
    },
    "id": "G6DPtYCEtPMy",
    "outputId": "94c1af3b-a574-45a0-f4c0-05a1ce85f6e6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How long can the coronavirus survive on surfaces?</td>\n",
       "      <td>On inanimate surfaces, human coronaviruses can remain infectious for up to 9 days.</td>\n",
       "      <td>0.984978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What means COVID-19?</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.927158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Is COVID19 worse than flue?</td>\n",
       "      <td>In comparison, COVID-19 showed similar trends with SARS patients  .</td>\n",
       "      <td>0.064450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>When the vaccine will be ready?</td>\n",
       "      <td>Current vaccine strategies take approximately 6 months for production.</td>\n",
       "      <td>0.210023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Whats the proteins that consist COVID-19?</td>\n",
       "      <td>CSFV contains 4 structural proteins: C, Erns, E1 and E2.</td>\n",
       "      <td>0.000411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Whats the symptoms of COVID-19?</td>\n",
       "      <td>This is particularly true for the COVID-19.</td>\n",
       "      <td>0.134529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>How can I prevent COVID-19?</td>\n",
       "      <td>It remains to be seen if this will be the case for COVID-19 as well.</td>\n",
       "      <td>0.246872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What treatments are available for COVID-19?</td>\n",
       "      <td>As effective drugs for SARS, hormones and interferons can also be used to treat COVID-19  .</td>\n",
       "      <td>0.956893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Is hand sanitizer effective against COVID-19?</td>\n",
       "      <td>The use of alcohol-based hand sanitizers is also effective [54, 55] .</td>\n",
       "      <td>0.205787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Am I at risk for serious complications from COVID-19 if I smoke cigarettes?</td>\n",
       "      <td>reported that people who have not been exposed to SARS-CoV-2 are all susceptible to COVID-19  .</td>\n",
       "      <td>0.015109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Are there any FDA-approved drugs (medicines) for COVID-19?</td>\n",
       "      <td>The new name is also not consistent with the disease name COVID-19.</td>\n",
       "      <td>0.015573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>How are people tested?</td>\n",
       "      <td>Student's t-tests were used to assess differences between groups.</td>\n",
       "      <td>0.009154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Why is the disease being called coronavirus disease 2019, COVID-19?</td>\n",
       "      <td>COVID-19: coronavirus disease 2019; PPE: personal protective equipment.</td>\n",
       "      <td>0.937286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Am I at risk for COVID-19 from mail, packages, or products?</td>\n",
       "      <td>This is particularly true for the COVID-19.</td>\n",
       "      <td>0.184955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>What is community spread?</td>\n",
       "      <td>Community spread and subclinical transmission need more attention.</td>\n",
       "      <td>0.235021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>How can I protect myself?</td>\n",
       "      <td>Get as much protection as you can.</td>\n",
       "      <td>0.050189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>What is a novel coronavirus?</td>\n",
       "      <td>Coronavirus.</td>\n",
       "      <td>0.111760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Was Harry Potter a good magician?</td>\n",
       "      <td>The vaccines elicited humoral immune responses and protective efficacy in mice  .</td>\n",
       "      <td>0.000183</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          Query  ... Cosine_similarity\n",
       "0                             How long can the coronavirus survive on surfaces?  ...          0.984978\n",
       "1                                                          What means COVID-19?  ...          0.927158\n",
       "2                                                   Is COVID19 worse than flue?  ...          0.064450\n",
       "3                                               When the vaccine will be ready?  ...          0.210023\n",
       "4                                     Whats the proteins that consist COVID-19?  ...          0.000411\n",
       "5                                               Whats the symptoms of COVID-19?  ...          0.134529\n",
       "6                                                   How can I prevent COVID-19?  ...          0.246872\n",
       "7                                   What treatments are available for COVID-19?  ...          0.956893\n",
       "8                                 Is hand sanitizer effective against COVID-19?  ...          0.205787\n",
       "9   Am I at risk for serious complications from COVID-19 if I smoke cigarettes?  ...          0.015109\n",
       "10                   Are there any FDA-approved drugs (medicines) for COVID-19?  ...          0.015573\n",
       "11                                                       How are people tested?  ...          0.009154\n",
       "12          Why is the disease being called coronavirus disease 2019, COVID-19?  ...          0.937286\n",
       "13                  Am I at risk for COVID-19 from mail, packages, or products?  ...          0.184955\n",
       "14                                                    What is community spread?  ...          0.235021\n",
       "15                                                    How can I protect myself?  ...          0.050189\n",
       "16                                                 What is a novel coronavirus?  ...          0.111760\n",
       "17                                            Was Harry Potter a good magician?  ...          0.000183\n",
       "\n",
       "[18 rows x 3 columns]"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with pd.option_context('display.max_colwidth', None):\n",
    "  display(myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n3Wun4hM40Bq"
   },
   "source": [
    "# References\n",
    "\n",
    "[1] https://colab.research.google.com/drive/1l6stpYdRMmeDBK_vw0L5NitdiAuhdsAr?usp=sharing#scrollTo=D_hDi8KzNgMM\n",
    "\n",
    "[2] https://www.sbert.net/docs/package_reference/cross_encoder.html"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "SBERT_CORD19_QA_CrossEncoders.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0ee152d36c8240f39ef86a0cbfbe1b64": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_83071c19219e403f97bd7d6ae6a291e9",
      "placeholder": "​",
      "style": "IPY_MODEL_5fb688f01e1d4d10970eb59d3ae8efba",
      "value": " 6793/6793 [02:13&lt;00:00, 50.72it/s]"
     }
    },
    "2513c9df07e64820a11655a80505a1f0": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "26aae27ca2154ef8929159efe55cbc7f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "3019714788cb417c81cf7f2ee30260eb": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5fb688f01e1d4d10970eb59d3ae8efba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "83071c19219e403f97bd7d6ae6a291e9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b4e5ac60951746dcbd8e263a6c1b3cbd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_dfdca8daf8784a3f9895461ba8789fe3",
       "IPY_MODEL_0ee152d36c8240f39ef86a0cbfbe1b64"
      ],
      "layout": "IPY_MODEL_2513c9df07e64820a11655a80505a1f0"
     }
    },
    "dfdca8daf8784a3f9895461ba8789fe3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Batches: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3019714788cb417c81cf7f2ee30260eb",
      "max": 6793,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_26aae27ca2154ef8929159efe55cbc7f",
      "value": 6793
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}