inferential_t5.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Vamsi995/Paraphrase-Generator/blob/master/Paraphrase.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8f6uVYL1XLJI"
      },
      "source": [
        "## **Inferential Transformer: Semantic Alignment Generation from Pre-trained Generative Model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "w2gJ5EV12trx"
      },
      "source": [
        "### **What models can be used**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-DBl7JPyjtKz"
      },
      "source": [
        "### T5 (Encoder decoder transformer)\n",
        "\n",
        "\n",
        "  T5 (*Text-to-text Transfer Transformer*) is yet another transformer model for Google Research Group. This is a vanilla transformer i.e this model has both the encoder and decoder layers as described in the paper [ Attention is All You Need](https://arxiv.org/abs/1706.03762). T5 is a encoder-decoder model that reaches SOTA results by solving NLP problems with a text-to-text approach. This is where text is used as both an input and an output for solving all types of tasks. This was introduced in the recent paper, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer ([paper](https://arxiv.org/abs/1910.10683)).\n",
        "\n",
        "  ![](https://1.bp.blogspot.com/-89OY3FjN0N0/XlQl4PEYGsI/AAAAAAAAFW4/knj8HFuo48cUFlwCHuU5feQ7yxfsewcAwCLcBGAsYHQ/s640/image2.png)\n",
        "\n",
        "  As we can see from the above picture, we have prepend our task name in front of the input and pass it to the model. If we want to fine tune our model to a specific task then our inputs should be modified to \"task_name: input_sentence < /s> \" and the output should be modified to \"output_sentence < /s>\". This model then can be trained on this modified input and output sequences.\n",
        "\n",
        "\n",
        "\n",
        "To explore this model:\n",
        "\n",
        "1. [T5 Paper](https://arxiv.org/abs/1910.10683)\n",
        "\n",
        "2. [A Brief Paper Analysis](https://towardsdatascience.com/t5-text-to-text-transformer-a-brief-paper-analysis-e4bba797bd68)\n",
        "\n",
        "3. [Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html)\n",
        "\n",
        "4. [Collin Rafel's Talk](https://www.youtube.com/watch?v=eKqWC577WlI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5u836VjQ2aPY"
      },
      "source": [
        "### Bart (Encoder decoder transformer)\n",
        "\n",
        "This is similar to the T5 model. This was released by the Facebook's AI team. To explore more about this model I suggest you to go through these links:\n",
        "\n",
        "1. [Blog Post](https://mc.ai/introducing-bart-combining-the-power-of-bert-and-gpt/)\n",
        "\n",
        "2. [Introducing BART](https://sshleifer.github.io/blog_v2/jupyter/2020/03/12/bart.html)\n",
        "\n",
        "3. [BART Paper](https://www.aclweb.org/anthology/2020.acl-main.703.pdf)\n",
        "\n",
        "4. [Hugging Face Docs](https://huggingface.co/transformers/model_doc/bart.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "a5h9vHxs2eKE"
      },
      "source": [
        "### Transformer + seq2seq model\n",
        "\n",
        "  The output from encoder models like BERT, is fed in to a seq2seq encoder-decoder model. To explore this model in detail read through the [paper](https://www.aclweb.org/anthology/D19-5627.pdf).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Zn-HWPP83z6Y"
      },
      "source": [
        "### Encoder-Decoder model\n",
        "\n",
        "  In this model, one BERT model is used as the encoder and the other BERT model is used as a decoder. Hugging face has a method to implement this model using the Encoder-Decoder Class where we can instantiate one model as an encoder and any other model as a decoder. As of now only the support of BERT2BERT models exist. You can head to the [docs](https://huggingface.co/transformers/model_doc/encoderdecoder.html) for futher implementation details. Do note that there is no Tensorflow implementation of this Encoder-Decoder class as of now only the Pytorch bindings exist.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "V__cOHPS2t4J"
      },
      "source": [
        "### **Building the T5 model for fine tuning**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Set Up Environemnt and Seed**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "colab_type": "code",
        "id": "f7izi0clHdsf",
        "outputId": "832db418-9aba-4f28-d6ef-c37ab85135d0"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import argparse\n",
        "import torch\n",
        "import logging \n",
        "import pandas as pd\n",
        "import pytorch_lightning as pl\n",
        "\n",
        "from transformers import AdamW, T5ForConditionalGeneration, T5Tokenizer, get_linear_schedule_with_warmup\n",
        "from saint.dataset.multi_task_dataset import MultiTaskDataset\n",
        "from saint.utils.utils import set_seed\n",
        "\n",
        "logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s',\n",
        "                    datefmt='%m/%d/%Y %H:%M:%S',\n",
        "                    level=logging.INFO)\n",
        "logger = logging.getLogger(__name__)\n",
        "\n",
        "set_seed(42)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\Users\\Admin\\anaconda3\\lib\\site-packages\\transformers\\models\\t5\\tokenization_t5.py:163: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.\n",
            "For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.\n",
            "- Be aware that you SHOULD NOT rely on t5-base automatically truncating your input to 512 when padding/encoding.\n",
            "- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.\n",
            "- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "tokenizer = T5Tokenizer.from_pretrained(\"t5-base\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "model = T5ForConditionalGeneration.from_pretrained(\"t5-base\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "4.019576072692871"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "input_ids = tokenizer(\"Given 'The lady is standing <extra_id_0> who are <extra_id_1>.' The conclusion 'The <extra_id_3> are <extra_id_4>' is ture\", return_tensors=\"pt\").input_ids\n",
        "labels = tokenizer(\"<extra_id_0> next to her two children <extra_id_1> eating a pizza <extra_id_2> The two children near the lady <extra_id_3> are eating something.\", return_tensors=\"pt\").input_ids\n",
        "\n",
        "# the forward function automatically creates the correct decoder_input_ids\n",
        "loss = model(input_ids=input_ids, labels=labels).loss\n",
        "loss.item()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Prepare Few-shot Dataset**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from saint.utils import py_io\n",
        "\n",
        "train_raw_data = py_io.read_jsonl(\"./train_snli.jsonl\")\n",
        "val_raw_data = py_io.read_jsonl(\"./test_snli.jsonl\")\n",
        "\n",
        "label_2_answer = {\n",
        "    \"entailment\": \"correct\",\n",
        "    \"contradiction\": \"incorrect\",\n",
        "    \"neutral\": \"inconclusive\"\n",
        "}\n",
        "\n",
        "train_data_nli = []\n",
        "val_data_nli = []\n",
        "train_data_evid = []\n",
        "val_data_evid = []\n",
        "train_data_enli = []\n",
        "val_data_enli = []\n",
        "\n",
        "def build_data_instance(data):\n",
        "    premise = data['premise']\n",
        "    hypothesis = data['hypothesis']\n",
        "    label = data['gold_label']\n",
        "    answer = label_2_answer[label]\n",
        "\n",
        "    evidence = [f\"{pair[0]}->{pair[1]}\" for pair in zip(data['Evidence1'], data['Evidence2'])]\n",
        "\n",
        "    nli_prompt = f\"nli: Given \\\"{premise}\\\" We should assume that \" + \\\n",
        "        f\"\\\"{hypothesis}\\\" is definitely definitely correct, incorrect, or inconclusive?\"\n",
        "    evid_prompt = f\"List evidence: Given \\\"{premise}\\\" We should assume that \" + \\\n",
        "        f\"\\\"{hypothesis}\\\" is definitely {answer} What is the evidence?\"\n",
        "    enli_prompt = f\"Given \\\"{premise}\\\" We should assume that \" + \\\n",
        "        f\"\\\"{hypothesis}\\\" is definitely correct, incorrect, or inconclusive? List evidence\"\n",
        "\n",
        "    nli_example = {\n",
        "        \"prompt\": nli_prompt,\n",
        "        \"output\": answer,\n",
        "    }\n",
        "\n",
        "    evid_example = {\n",
        "        \"prompt\": evid_prompt,\n",
        "        \"output\": \"; \".join(evidence)\n",
        "    }\n",
        "\n",
        "    enli_example = {\n",
        "        \"prompt\": enli_prompt,\n",
        "        \"output\": f\"{answer} supported by this list of evidence: \" + \"; \".join(evidence)\n",
        "    }\n",
        "    \n",
        "    return nli_example, evid_example, enli_example\n",
        "\n",
        "def write_all_input_data(task):\n",
        "    py_io.write_jsonl(train_data_nli, f\"data/{task}/nli_train.jsonl\")\n",
        "    py_io.write_jsonl(train_data_evid, f\"data/{task}/evid_train.jsonl\")\n",
        "    py_io.write_jsonl(train_data_enli, f\"data/{task}/enli_train.jsonl\")\n",
        "    py_io.write_jsonl(val_data_nli, f\"data/{task}/nli_val.jsonl\")\n",
        "    py_io.write_jsonl(val_data_evid, f\"data/{task}/evid_val.jsonl\")\n",
        "    py_io.write_jsonl(val_data_enli, f\"data/{task}/enli_val.jsonl\")\n",
        "\n",
        "for data in train_raw_data:\n",
        "    nli_example, evid_example, enli_example = build_data_instance(data)\n",
        "    train_data_nli.append(nli_example)\n",
        "    train_data_evid.append(evid_example)\n",
        "    train_data_enli.append(enli_example)\n",
        "\n",
        "for data in val_raw_data:\n",
        "    nli_example, evid_example, enli_example = build_data_instance(data)\n",
        "    val_data_nli.append(nli_example)\n",
        "    val_data_evid.append(evid_example)\n",
        "    val_data_enli.append(enli_example)\n",
        "\n",
        "write_all_input_data(\"snli\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Multi-task Dataset Setup**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "import copy\n",
        "import re\n",
        "import string\n",
        "import torch\n",
        "import numpy as np\n",
        "import saint.utils.py_io as py_io\n",
        "\n",
        "from collections import Counter\n",
        "from torch.utils.data import (\n",
        "    Dataset,\n",
        "    DataLoader,\n",
        "    RandomSampler,\n",
        "    SequentialSampler\n",
        ")\n",
        "\n",
        "\n",
        "class QADataset(Dataset):\n",
        "    def __init__(self, inputs_tokenized, outputs_tokenized):\n",
        "        \n",
        "        self.inputs_tokenized = inputs_tokenized\n",
        "        self.outputs_tokenized = outputs_tokenized\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.inputs_tokenized)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        source_ids = self.inputs_tokenized[index][\"input_ids\"].squeeze()\n",
        "        target_ids = self.outputs_tokenized[index][\"input_ids\"].squeeze()\n",
        "\n",
        "        src_mask = self.inputs_tokenized[index][\"attention_mask\"].squeeze()\n",
        "        target_mask = self.outputs_tokenized[index][\"attention_mask\"].squeeze()\n",
        "\n",
        "        labels = copy.deepcopy(target_ids)\n",
        "        labels[labels == 0] = -100\n",
        "\n",
        "        return {\n",
        "            \"source_ids\": source_ids,\n",
        "            \"source_mask\": src_mask,\n",
        "            \"target_ids\": target_ids,\n",
        "            \"target_mask\": target_mask,\n",
        "            \"labels\": labels\n",
        "        }\n",
        "\n",
        "\n",
        "class MultiTaskDataLoader(DataLoader):\n",
        "    def __init__(self, dataset, is_training, \n",
        "                 train_batch_size, val_batch_size):\n",
        "        if is_training:\n",
        "            sampler = RandomSampler(dataset)\n",
        "            batch_size = train_batch_size\n",
        "        else:\n",
        "            sampler = SequentialSampler(dataset)\n",
        "            batch_size = val_batch_size\n",
        "        super(MultiTaskDataLoader, self).__init__(\n",
        "            dataset, sampler=sampler, batch_size=batch_size)\n",
        "\n",
        "\n",
        "class MultiTaskDataset:\n",
        "\n",
        "    def __init__(self, logger, data_path, tasks, data_split, train_batch_size, val_batch_size, is_training):\n",
        "        self.data_path = data_path\n",
        "        self.data = []\n",
        "\n",
        "        for task in sorted(tasks):\n",
        "            task_dir = os.path.join(self.data_path, task)\n",
        "            \n",
        "            prefixes = [\"nli\", \"evid\"]\n",
        "\n",
        "            for prefix in prefixes:\n",
        "                train_file_pth = os.path.join(\n",
        "                    task_dir, f\"{prefix}_train.jsonl\")\n",
        "                train_examples = py_io.read_jsonl(path=train_file_pth)\n",
        "\n",
        "                dev_file_pth = os.path.join(\n",
        "                    task_dir, f\"{prefix}_val.jsonl\")\n",
        "                dev_examples = py_io.read_jsonl(path=dev_file_pth)\n",
        "\n",
        "                self.data.append({\n",
        "                    \"task_name\": task,\n",
        "                    \"task_prefix\": prefix,\n",
        "                    \"train_examples\": train_examples,\n",
        "                    \"dev_examples\": dev_examples\n",
        "                })\n",
        "\n",
        "        self.data_split = data_split\n",
        "        self.is_training = is_training\n",
        "        self.logger = logger\n",
        "\n",
        "        self.metric = \"EM\"\n",
        "        self.tokenizer = None\n",
        "        self.dataset = None\n",
        "        self.dataloader = None\n",
        "        self.cache = None\n",
        "        self.load = True\n",
        "        self.gen_early_stop = False\n",
        "        self.max_len_input = 128\n",
        "        self.max_len_output = 128\n",
        "        self.train_batch_size = train_batch_size\n",
        "        self.val_batch_size = val_batch_size\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.data)\n",
        "\n",
        "    def decode(self, tokens):\n",
        "        return self.tokenizer.decode(tokens, skip_special_tokens=True, clean_up_tokenization_spaces=True)\n",
        "\n",
        "    def decode_batch(self, tokens):\n",
        "        return [self.decode(_tokens) for _tokens in tokens]\n",
        "\n",
        "    def _build_cache_path(self, tokenizer):\n",
        "        postfix = tokenizer.__class__.__name__.replace(\"zer\", \"zed\")\n",
        "        return os.path.join(\n",
        "            self.data_path,\n",
        "            f\"{self.data_split}-multi-{postfix}.pt\"\n",
        "        )\n",
        "\n",
        "    def load_dataset(self, tokenizer, do_return=False):\n",
        "        self.tokenizer = tokenizer\n",
        "        preprocessed_path = self._build_cache_path(tokenizer)\n",
        "\n",
        "        if self.load and os.path.exists(preprocessed_path):\n",
        "            self.logger.info(\n",
        "                f\"Loading pre-tokenized data from {preprocessed_path}\")\n",
        "            tokenized = torch.load(preprocessed_path)\n",
        "            inputs_tokenized = tokenized[\"inputs_tokenized\"]\n",
        "            outputs_tokenized = tokenized[\"outputs_tokenized\"]\n",
        "        else:\n",
        "            self.logger.info(\n",
        "                f\"Start tokenizing ... {len(self.data)} instances\")\n",
        "\n",
        "            inputs = []\n",
        "            outputs = []\n",
        "            inputs_tokenized = []\n",
        "            outputs_tokenized = []\n",
        "\n",
        "            for task in self.data:\n",
        "                if self.data_split in [\"train\", \"all\"]:\n",
        "                    for dp in task[\"train_examples\"]:\n",
        "                        inputs.append(dp[\"prompt\"])\n",
        "                        outputs.append(dp[\"output\"])\n",
        "                if self.data_split in [\"val\", \"all\"]:\n",
        "                    for dp in task[\"dev_examples\"]:\n",
        "                        inputs.append(dp[\"prompt\"])\n",
        "                        outputs.append(dp[\"output\"])\n",
        "\n",
        "            self.logger.info(\"Printing 3 examples\")\n",
        "            for i in range(3):\n",
        "                self.logger.info(inputs[i])\n",
        "                self.logger.info(outputs[i])\n",
        "\n",
        "            # if self.args.do_lowercase:\n",
        "            #     inputs = [input0.lower() for input0 in inputs]\n",
        "            #     outputs = [output0.lower() for output0 in outputs]\n",
        "            # if self.args.append_another_bos:\n",
        "            #     inputs = [f\"<s> {input0}\" for input0 in inputs]\n",
        "            #     outputs = [f\"<s> {output0}\" for output0 in outputs]\n",
        "\n",
        "            self.logger.info(\"Tokenizing data points ...\")\n",
        "            for prompt, output in zip(inputs, outputs):\n",
        "                tokenized_inputs = self.tokenizer.batch_encode_plus(\n",
        "                    [prompt], \n",
        "                    max_length=self.max_len_input, \n",
        "                    pad_to_max_length=True, \n",
        "                    return_tensors=\"pt\"\n",
        "                )\n",
        "\n",
        "                tokenized_targets = self.tokenizer.batch_encode_plus(\n",
        "                    [output], \n",
        "                    max_length=self.max_len_output, \n",
        "                    pad_to_max_length=True, \n",
        "                    return_tensors=\"pt\"\n",
        "                )\n",
        "\n",
        "                inputs_tokenized.append(tokenized_inputs)\n",
        "                outputs_tokenized.append(tokenized_targets)\n",
        "\n",
        "            if self.load:\n",
        "                preprocessed_data = {\n",
        "                    \"inputs_tokenized\": inputs_tokenized,\n",
        "                    \"outputs_tokenized\": outputs_tokenized\n",
        "                }\n",
        "                torch.save(preprocessed_data, preprocessed_path)\n",
        "\n",
        "        self.dataset = QADataset(inputs_tokenized, outputs_tokenized)\n",
        "        self.logger.info(\n",
        "            f\"Loaded {len(self.dataset)} examples from {self.data_split} data\")\n",
        "\n",
        "        if do_return:\n",
        "            return self.dataset\n",
        "\n",
        "    def load_dataloader(self, do_return=False):\n",
        "        self.dataloader = MultiTaskDataLoader(\n",
        "            self.dataset, self.is_training,\n",
        "            self.train_batch_size, self.val_batch_size\n",
        "        )\n",
        "        if do_return:\n",
        "            return self.dataloader\n",
        "\n",
        "    def save_predictions(self, predictions):\n",
        "        assert len(predictions) == len(self), (len(predictions), len(self))\n",
        "\n",
        "        predictions = [\n",
        "            'n/a' if len(prediction.strip()) ==\n",
        "            0 else prediction for prediction in predictions\n",
        "        ]\n",
        "        prediction_text = [\n",
        "            prediction.strip()+'\\n' for prediction in predictions]\n",
        "        save_path = os.path.join(\n",
        "            self.output_dir,\n",
        "            f\"{self.prefix}_predictions.txt\")\n",
        "        with open(save_path, \"w\") as f:\n",
        "            f.writelines(prediction_text)\n",
        "\n",
        "        self.logger.info(f\"Saved prediction in {save_path}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "03/06/2022 13:48:13 - INFO - __main__ - Start tokenizing ... 2 instances\n",
            "03/06/2022 13:48:13 - INFO - __main__ - Printing 3 examples\n",
            "03/06/2022 13:48:13 - INFO - __main__ - nli: Given \"A person dressed in a coat and hat holds a tray on his bicycle.\" We should assume that \"A person is travelling\" is definitely definitely correct, incorrect, or inconclusive?\n",
            "03/06/2022 13:48:13 - INFO - __main__ - inconclusive\n",
            "03/06/2022 13:48:13 - INFO - __main__ - nli: Given \"A man in a helmet is riding a bike.\" We should assume that \"The man in a helmet is riding a sports bike.\" is definitely definitely correct, incorrect, or inconclusive?\n",
            "03/06/2022 13:48:13 - INFO - __main__ - inconclusive\n",
            "03/06/2022 13:48:13 - INFO - __main__ - nli: Given \"Two workers are cleaning a window with bystanders and a dog in the background.\" We should assume that \"A dog plays fetch near two workers.\" is definitely definitely correct, incorrect, or inconclusive?\n",
            "03/06/2022 13:48:13 - INFO - __main__ - inconclusive\n",
            "03/06/2022 13:48:13 - INFO - __main__ - Tokenizing data points ...\n",
            "Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n",
            "C:\\Users\\Admin\\anaconda3\\lib\\site-packages\\transformers\\tokenization_utils_base.py:2251: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  warnings.warn(\n",
            "03/06/2022 13:48:13 - INFO - __main__ - Loaded 600 examples from train data\n",
            "03/06/2022 13:48:13 - INFO - __main__ - Start tokenizing ... 2 instances\n",
            "03/06/2022 13:48:13 - INFO - __main__ - Printing 3 examples\n",
            "03/06/2022 13:48:13 - INFO - __main__ - nli: Given \"A group of people gathered at night watching an event.\" We should assume that \"A group of friends are watching the music concert at night.\" is definitely definitely correct, incorrect, or inconclusive?\n",
            "03/06/2022 13:48:13 - INFO - __main__ - inconclusive\n",
            "03/06/2022 13:48:13 - INFO - __main__ - nli: Given \"Children are playing in the street with a toy.\" We should assume that \"The children have a ball.\" is definitely definitely correct, incorrect, or inconclusive?\n",
            "03/06/2022 13:48:13 - INFO - __main__ - inconclusive\n",
            "03/06/2022 13:48:13 - INFO - __main__ - nli: Given \"A boy swims shirtless with a black and red bathing suit while a seagull sits on the sand.\" We should assume that \"A boy swims while a seagull shits on the sand.\" is definitely definitely correct, incorrect, or inconclusive?\n",
            "03/06/2022 13:48:13 - INFO - __main__ - inconclusive\n",
            "03/06/2022 13:48:13 - INFO - __main__ - Tokenizing data points ...\n",
            "03/06/2022 13:48:13 - INFO - __main__ - Loaded 198 examples from val data\n"
          ]
        }
      ],
      "source": [
        "#from saint.multi_task_dataset import MultiTaskDataset\n",
        "\n",
        "model_name_or_path = \"t5-base\"\n",
        "tokenizer = T5Tokenizer.from_pretrained(model_name_or_path)\n",
        "\n",
        "data_dir = \"data\"\n",
        "\n",
        "train_multi_data = MultiTaskDataset(\n",
        "    logger, data_path=data_dir, \n",
        "    tasks=[\"snli\"], data_split=\"train\", \n",
        "    train_batch_size=2, val_batch_size=2,\n",
        "    is_training=True)\n",
        "dev_multi_data = MultiTaskDataset(\n",
        "    logger, data_path=data_dir, \n",
        "    tasks=[\"snli\"], data_split=\"val\",\n",
        "    train_batch_size=2, val_batch_size=2, \n",
        "    is_training=False)\n",
        "\n",
        "train_multi_data.load_dataset(tokenizer)\n",
        "train_multi_data.load_dataloader()\n",
        "\n",
        "dev_multi_data.load_dataset(tokenizer)\n",
        "dev_multi_data.load_dataloader()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Create Fine-tuner and Model**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [],
      "source": [
        "class MetricsCallback(pl.Callback):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.metrics = []\n",
        "\n",
        "    def on_validation_end(self, trainer, pl_module):\n",
        "        self.metrics.append(trainer.callback_metrics)\n",
        "\n",
        "class T5FineTuner(pl.LightningModule):\n",
        "    def __init__(self, args, train_dataset, val_dataset):\n",
        "        super(T5FineTuner, self).__init__()\n",
        "        self.save_hyperparameters(args)\n",
        "        self.args = args\n",
        "        self.train_dataset = train_dataset\n",
        "        self.val_dataset = val_dataset\n",
        "        self.tokenizer = T5Tokenizer.from_pretrained(\n",
        "            args.model_name_or_path)\n",
        "        self.model = T5ForConditionalGeneration.from_pretrained(\n",
        "            args.model_name_or_path)\n",
        "        \n",
        "\n",
        "    def forward(self, input_ids,\n",
        "                attention_mask=None,\n",
        "                decoder_input_ids=None,\n",
        "                decoder_attention_mask=None,\n",
        "                lm_labels=None):\n",
        "        outputs = self.model(\n",
        "            input_ids=input_ids,\n",
        "            attention_mask=attention_mask,\n",
        "            decoder_attention_mask=decoder_attention_mask,\n",
        "            labels=lm_labels,\n",
        "        )\n",
        "\n",
        "        return outputs\n",
        "\n",
        "    def training_step(self, batch, batch_idx):\n",
        "        outputs = self.forward(\n",
        "            input_ids=batch[\"source_ids\"],\n",
        "            attention_mask=batch[\"source_mask\"],\n",
        "            decoder_input_ids = batch[\"target_ids\"],\n",
        "            decoder_attention_mask=batch[\"target_mask\"],\n",
        "            lm_labels=batch[\"labels\"]\n",
        "        )\n",
        "\n",
        "        loss = outputs[0]\n",
        "        self.log('train_loss',loss)\n",
        "        return loss\n",
        "\n",
        "    def validation_step(self, batch, batch_idx):\n",
        "        outputs = self.forward(\n",
        "            input_ids=batch[\"source_ids\"],\n",
        "            attention_mask=batch[\"source_mask\"],\n",
        "            decoder_input_ids = batch[\"target_ids\"],\n",
        "            decoder_attention_mask=batch[\"target_mask\"],\n",
        "            lm_labels=batch[\"labels\"]\n",
        "        )\n",
        "\n",
        "        loss = outputs[0]\n",
        "        self.log(\"val_loss\",loss)\n",
        "        return loss\n",
        "\n",
        "    def save_core_model(self):\n",
        "        store_path = os.path.join(\n",
        "            self.args.output_dir,\n",
        "            self.args.name)\n",
        "        self.model.save_pretrained(store_path)\n",
        "        self.tokenizer.save_pretrained(store_path)\n",
        "\n",
        "    def configure_optimizers(self):\n",
        "        optimizer = AdamW(self.parameters(), lr=self.hparams.lr, eps=1e-8)\n",
        "        scheduler = get_linear_schedule_with_warmup(\n",
        "            optimizer, num_warmup_steps=self.hparams.warmup_steps,\n",
        "            num_training_steps=self.hparams.max_epochs * len(self.train_dataset))\n",
        "        return {'optimizer': optimizer, 'lr_scheduler': scheduler}\n",
        "\n",
        "    def train_dataloader(self):\n",
        "        return self.train_dataset.dataloader\n",
        "\n",
        "    def val_dataloader(self):\n",
        "        return self.val_dataset.dataloader"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### **Fine-tune Your Own Inferential T5**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Initialize Hyperparameters and Training Arguments**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "import argparse\n",
        "\n",
        "trial_number = 1\n",
        "\n",
        "args_dict = dict(\n",
        "    name=\"snli-evidence-t5-base\",\n",
        "    data_dir=\"\",\n",
        "    output_dir=\"./runs\",\n",
        "    model_name_or_path='t5-base',\n",
        "    max_seq_length=512,\n",
        "    lr=3e-5,\n",
        "    weight_decay=0.0,\n",
        "    warmup_steps=0,\n",
        "    train_batch_size=2,\n",
        "    eval_batch_size=2,\n",
        "    max_epochs=7,\n",
        "    gradient_accumulation_steps=16,\n",
        "    n_gpu=1,\n",
        "    fp_16=False,\n",
        "    opt_level='O1',\n",
        "    max_grad_norm=1.0,\n",
        "    seed=42,\n",
        ")\n",
        "\n",
        "args = argparse.Namespace(**args_dict)\n",
        "\n",
        "checkpoint_callback = pl.callbacks.ModelCheckpoint(\n",
        "    dirpath=os.path.join(args.output_dir, args.name),\n",
        "    filename=\"checkpoint_best\",\n",
        "    monitor=\"val_loss\",\n",
        "    mode=\"min\",\n",
        "    save_top_k=2\n",
        ")\n",
        "\n",
        "metrics_callback = MetricsCallback()\n",
        "\n",
        "train_params = dict(\n",
        "    #accumulate_grad_batches=args.gradient_accumulation_steps,\n",
        "    gpus=args.n_gpu,\n",
        "    max_epochs=args.max_epochs,\n",
        "    progress_bar_refresh_rate=10,\n",
        "    #precision= 16 if args.fp_16 else 32,\n",
        "    #amp_level=args.opt_level,\n",
        "    #gradient_clip_val=args.max_grad_norm,\n",
        "    checkpoint_callback=True,\n",
        "    callbacks=[metrics_callback, checkpoint_callback]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Run Training and Validation Loop**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "GPU available: True, used: True\n",
            "TPU available: False, using: 0 TPU cores\n",
            "IPU available: False, using: 0 IPUs\n",
            "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " Training model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "  | Name  | Type                       | Params\n",
            "-----------------------------------------------------\n",
            "0 | model | T5ForConditionalGeneration | 222 M \n",
            "-----------------------------------------------------\n",
            "222 M     Trainable params\n",
            "0         Non-trainable params\n",
            "222 M     Total params\n",
            "891.614   Total estimated model params size (MB)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 6: 100%|██████████| 399/399 [00:38<00:00, 10.29it/s, loss=0.328, v_num=13]\n",
            "Saving model ... \n",
            "Model Saved\n",
            "C:\\Users\\Admin\\Desktop\\Research-spring-2022\\ultralog\\runs\\snli-evidence-t5-base\\checkpoint_best.ckpt\n"
          ]
        }
      ],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "model = T5FineTuner(args, train_dataset=train_multi_data, val_dataset=dev_multi_data)\n",
        "trainer = pl.Trainer(**train_params)\n",
        "\n",
        "print (\" Training model\")\n",
        "trainer.fit(model)\n",
        "#trainer.test(model)\n",
        "\n",
        "print (\"Saving model ... \")\n",
        "model.save_core_model()\n",
        "\n",
        "print(\"Model Saved\")\n",
        "print(checkpoint_callback.best_model_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### **Test On Validation Set**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [],
      "source": [
        "from tqdm import tqdm\n",
        "from saint.utils.metric import (\n",
        "    compute_f1,\n",
        "    compute_rouge,\n",
        "    compute_bleu_score,\n",
        "    compute_exact_match\n",
        ")\n",
        "\n",
        "import torch\n",
        "\n",
        "from transformers import T5ForConditionalGeneration, T5Tokenizer\n",
        "\n",
        "class T5AlignmentGenerator:\n",
        "\n",
        "    def __init__(self, model, test_data):\n",
        "        self.model = T5ForConditionalGeneration.from_pretrained(model)\n",
        "        self.tokenizer = T5Tokenizer.from_pretrained(model)\n",
        "\n",
        "        self.model.eval()\n",
        "        self.model.cuda()\n",
        "\n",
        "        self.test_data = test_data\n",
        "\n",
        "    def _tokenization(self, sentences):\n",
        "        tokenized = self.tokenizer.encode_plus(\n",
        "            sentences, return_tensors=\"pt\")\n",
        "        tokenized.to('cuda')\n",
        "\n",
        "        input_ids = tokenized[\"input_ids\"]\n",
        "        attention_mask = tokenized[\"attention_mask\"]\n",
        "        return input_ids, attention_mask\n",
        "\n",
        "    def generate(self):\n",
        "        generation_output = []\n",
        "        with torch.no_grad():\n",
        "            for _, test_example in tqdm(enumerate(self.test_data)):\n",
        "                prompt = test_example['input']\n",
        "                target = test_example[\"output\"]\n",
        "                input_ids, attention_mask = self._tokenization(prompt)\n",
        "\n",
        "                beam_outputs = self.model.generate(\n",
        "                    input_ids=input_ids,\n",
        "                    attention_mask=attention_mask,\n",
        "                    max_length=50,\n",
        "                    num_beams=10,\n",
        "                    top_k=10,\n",
        "                    do_sample=True,\n",
        "                    early_stopping=True,\n",
        "                    num_return_sequences=5,\n",
        "                    no_repeat_ngram_size=2\n",
        "                )\n",
        "\n",
        "                for beam_output in beam_outputs:\n",
        "                    prediction = self.tokenizer.decode(\n",
        "                        beam_output,\n",
        "                        skip_special_tokens=True,\n",
        "                        clean_up__tokenization_spaces=True)\n",
        "                    f1_score = compute_f1(prediction, target)\n",
        "                    em_score = compute_exact_match(prediction, target)\n",
        "                    rouge_scores = compute_rouge(prediction, target)\n",
        "                    bleu_score = compute_bleu_score(prediction, target)\n",
        "\n",
        "                    result = {\n",
        "                        \"prompt\": prompt,\n",
        "                        \"prediction\": prediction,\n",
        "                        \"truth\": target,\n",
        "                        \"F1\": f1_score,\n",
        "                        \"EM\": em_score,\n",
        "                        \"BLEU-4\": bleu_score,\n",
        "                        \"ROUGEL\": rouge_scores['rougeL'][1],\n",
        "                    }\n",
        "                    generation_output.append(result)\n",
        "                    if rouge_scores['rougeL'][1] == 1:\n",
        "                        break\n",
        "        return generation_output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "1it [00:06,  6.83s/it]c:\\Users\\Admin\\anaconda3\\lib\\site-packages\\nltk\\translate\\bleu_score.py:516: UserWarning: \n",
            "The hypothesis contains 0 counts of 3-gram overlaps.\n",
            "Therefore the BLEU score evaluates to 0, independently of\n",
            "how many N-gram overlaps of lower order it contains.\n",
            "Consider using lower n-gram order or use SmoothingFunction()\n",
            "  warnings.warn(_msg)\n",
            "c:\\Users\\Admin\\anaconda3\\lib\\site-packages\\nltk\\translate\\bleu_score.py:516: UserWarning: \n",
            "The hypothesis contains 0 counts of 4-gram overlaps.\n",
            "Therefore the BLEU score evaluates to 0, independently of\n",
            "how many N-gram overlaps of lower order it contains.\n",
            "Consider using lower n-gram order or use SmoothingFunction()\n",
            "  warnings.warn(_msg)\n",
            "42it [00:14,  6.82it/s]c:\\Users\\Admin\\anaconda3\\lib\\site-packages\\nltk\\translate\\bleu_score.py:516: UserWarning: \n",
            "The hypothesis contains 0 counts of 2-gram overlaps.\n",
            "Therefore the BLEU score evaluates to 0, independently of\n",
            "how many N-gram overlaps of lower order it contains.\n",
            "Consider using lower n-gram order or use SmoothingFunction()\n",
            "  warnings.warn(_msg)\n",
            "75it [00:22,  3.33it/s]\n"
          ]
        }
      ],
      "source": [
        "from saint.utils import py_io\n",
        "\n",
        "dataset = \"causal_snli\"\n",
        "task = \"causal_snli_entail\"\n",
        "\n",
        "test_data = py_io.read_jsonl(f\"data/{dataset}/{task}/dev.jsonl\")\n",
        "t5_saint_generator = T5AlignmentGenerator(\"allenai/unifiedqa-t5-small\", test_data)\n",
        "alignment_output = t5_saint_generator.generate()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "375"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(alignment_output)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'avg_bleu': 0.4145414735197152, 'avg_rougeL_recall': 0.3281853281853282}"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from saint.utils.py_io import read_json\n",
        "\n",
        "relational_pred = alignment_output\n",
        "rougeL_recall_sum = 0\n",
        "for i in range(0, len(relational_pred)-5, 5):\n",
        "  r1 = relational_pred[i]['ROUGEL']\n",
        "  r2 = relational_pred[i+1]['ROUGEL']\n",
        "  r3 = relational_pred[i+2]['ROUGEL']\n",
        "  r4 = relational_pred[i+3]['ROUGEL']\n",
        "  r5 = relational_pred[i+4]['ROUGEL']\n",
        "  rougeL_recall_sum += max(r1, r2, r3, r4, r5)\n",
        "avg_rougeL_recall = rougeL_recall_sum / (len(relational_pred) / 5)\n",
        "\n",
        "bleu_sum = 0\n",
        "for i in range(0, len(relational_pred)-5, 5):\n",
        "  r1 = relational_pred[i]['BLEU-4']\n",
        "  r2 = relational_pred[i+1]['BLEU-4']\n",
        "  r3 = relational_pred[i+2]['BLEU-4']\n",
        "  r4 = relational_pred[i+3]['BLEU-4']\n",
        "  r5 = relational_pred[i+4]['BLEU-4']\n",
        "  bleu_sum += max(r1, r2, r3, r4, r5)\n",
        "avg_bleu = bleu_sum / (len(relational_pred) / 5)\n",
        "\n",
        "{\n",
        "  'avg_bleu': avg_bleu,\n",
        "  'avg_rougeL_recall': avg_rougeL_recall\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "relational_pred = read_json(\"relational_pred.json\")\n",
        "\n",
        "rougeL_recall_sum = 0\n",
        "for i in range(0, len(relational_pred)-5, 5):\n",
        "  r1 = relational_pred[i]['ROUGEL']\n",
        "  r2 = relational_pred[i+1]['ROUGEL']\n",
        "  r3 = relational_pred[i+2]['ROUGEL']\n",
        "  r4 = relational_pred[i+3]['ROUGEL']\n",
        "  r5 = relational_pred[i+4]['ROUGEL']\n",
        "  rougeL_recall_sum += max(r1, r2, r3, r4, r5)\n",
        "avg_rougeL_recall = rougeL_recall_sum / (len(relational_pred) / 5)\n",
        "\n",
        "bleu_sum = 0\n",
        "for i in range(0, len(relational_pred)-5, 5):\n",
        "  r1 = relational_pred[i]['BLEU-4']\n",
        "  r2 = relational_pred[i+1]['BLEU-4']\n",
        "  r3 = relational_pred[i+2]['BLEU-4']\n",
        "  r4 = relational_pred[i+3]['BLEU-4']\n",
        "  r5 = relational_pred[i+4]['BLEU-4']\n",
        "  bleu_sum += max(r1, r2, r3, r4, r5)\n",
        "avg_bleu = bleu_sum / (len(relational_pred) / 5)\n",
        "\n",
        "{\n",
        "  'avg_bleu': avg_bleu,\n",
        "  'avg_rougeL_recall': avg_rougeL_recall\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "ret_val = [{\n",
        "    \"premise\": \"Herceptin was already approved to treat the sickest breast cancer patients, and the company said, Monday, it will discuss with federal regulators the possibility of prescribing the drug for more breast cancer patients.\", \n",
        "    \"source\": \"Herceptin is used to treat breast cancer\",\n",
        "    \"target\": \"Herceptin was already approved to treat the sickest breast cancer patients\",\n",
        "  },\n",
        "  {\n",
        "    \"premise\": \"Herceptin was already approved to treat the sickest breast cancer patients, and the company said, Monday, it will discuss with federal regulators the possibility of prescribing the drug for more breast cancer patients.\", \n",
        "    \"source\": \"more breast cancer patients may get prescription of Herceptin\",\n",
        "    \"target\": \"the possibility of prescribing the drug for more breast cancer patients\",\n",
        "  },\n",
        "  {\n",
        "    \"premise\": \"Judie Vivian, chief executive at ProMedica, a medical service company that helps sustain the 2-year-old Vietnam Heart Institute in Ho Chi Minh City (formerly Saigon), said that so far about 1,500 children have received treatment.\",\n",
        "    \"source\": \"The previous name of Ho Chi Minh City was Saigon.\",\n",
        "    \"target\": \"Ho Chi Minh City (formerly Saigon)\",\n",
        "  }\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "premise = ret_val[0]['premise']\n",
        "source = ret_val[0]['source']\n",
        "target = ret_val[0][\"target\"]\n",
        "\n",
        "test_sent = f\"Premise: {premise} <extra_id_0> Source Phrase: {source} <extra_id_1> In Premise , Source Phrase aligns to: </s>\"\n",
        "test_tokenized = infer_model.tokenizer.encode_plus(test_sent, return_tensors=\"pt\")\n",
        "\n",
        "test_input_ids  = test_tokenized[\"input_ids\"]\n",
        "test_attention_mask = test_tokenized[\"attention_mask\"]\n",
        "\n",
        "infer_model.model.eval()\n",
        "beam_outputs = infer_model.model.generate(\n",
        "    input_ids=test_input_ids,\n",
        "    attention_mask=test_attention_mask,\n",
        "    max_length=64,\n",
        "    early_stopping=True,\n",
        "    num_beams=10,\n",
        "    num_return_sequences=3,\n",
        "    no_repeat_ngram_size=2\n",
        ")\n",
        "\n",
        "print(\"premise: \" + premise)\n",
        "print(\"source: \" + source)\n",
        "print(\"target: \" + target)\n",
        "\n",
        "print(\"Predictions: \")\n",
        "for beam_output in beam_outputs:\n",
        "    sent = infer_model.tokenizer.decode(beam_output, skip_special_tokens=True,clean_up_tokenization_spaces=True)\n",
        "    print (sent)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "premise = ret_val[1]['premise']\n",
        "source = ret_val[1]['source']\n",
        "target = \"the company said, Monday, it will discuss with federal regulators\"\n",
        "source = \"the company and federal regulators will have a discussion\"\n",
        "\n",
        "test_sent = f\"Premise: {premise} <extra_id_0> Source Phrase: {source} <extra_id_1> In Premise , Source Phrase aligns to: </s>\"\n",
        "test_tokenized = infer_model.tokenizer.encode_plus(test_sent, return_tensors=\"pt\")\n",
        "\n",
        "test_input_ids  = test_tokenized[\"input_ids\"]\n",
        "test_attention_mask = test_tokenized[\"attention_mask\"]\n",
        "\n",
        "infer_model.model.eval()\n",
        "beam_outputs = infer_model.model.generate(\n",
        "    input_ids=test_input_ids,\n",
        "    attention_mask=test_attention_mask,\n",
        "    max_length=20,\n",
        "    early_stopping=True,\n",
        "    num_beams=10,\n",
        "    num_return_sequences=3,\n",
        "    no_repeat_ngram_size=2\n",
        ")\n",
        "\n",
        "print(\"premise: \" + premise)\n",
        "print(\"source: \" + source)\n",
        "print(\"target: \" + target)\n",
        "\n",
        "print(\"Predictions: \")\n",
        "for beam_output in beam_outputs:\n",
        "    sent = infer_model.tokenizer.decode(beam_output, skip_special_tokens=True,clean_up_tokenization_spaces=True)\n",
        "    print (sent)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "premise = ret_val[2]['premise']\n",
        "source = ret_val[2]['source']\n",
        "target = ret_val[2][\"target\"]\n",
        "\n",
        "test_sent = f\"Premise: {premise} <extra_id_0> Source Phrase: {source} <extra_id_1> In Premise , Source Phrase aligns to: </s>\"\n",
        "test_tokenized = infer_model.tokenizer.encode_plus(test_sent, return_tensors=\"pt\")\n",
        "\n",
        "test_input_ids  = test_tokenized[\"input_ids\"]\n",
        "test_attention_mask = test_tokenized[\"attention_mask\"]\n",
        "\n",
        "infer_model.model.eval()\n",
        "beam_outputs = infer_model.model.generate(\n",
        "    input_ids=test_input_ids,\n",
        "    attention_mask=test_attention_mask,\n",
        "    max_length=20,\n",
        "    early_stopping=True,\n",
        "    num_beams=10,\n",
        "    num_return_sequences=3,\n",
        "    no_repeat_ngram_size=2\n",
        ")\n",
        "\n",
        "print(\"premise: \" + premise)\n",
        "print(\"source: \" + source)\n",
        "print(\"target: \" + target)\n",
        "\n",
        "print(\"Predictions: \")\n",
        "for beam_output in beam_outputs:\n",
        "    sent = infer_model.tokenizer.decode(beam_output, skip_special_tokens=True,clean_up_tokenization_spaces=True)\n",
        "    print (sent)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\Users\\Admin\\anaconda3\\envs\\causal_lens\\lib\\site-packages\\tqdm\\auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n",
            "Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaModel: ['classifier.dense.bias', 'classifier.out_proj.bias', 'classifier.out_proj.weight', 'classifier.dense.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ],
      "source": [
        "from saint.eval.auto_eval.ctc_scorer.bert_aligner import BERTAligner, E_MODEL_CONFIGS\n",
        "\n",
        "aligner = BERTAligner(\n",
        "    **E_MODEL_CONFIGS[\"roberta-mnli\"],\n",
        "    aggr_type=\"mean\",\n",
        "    lang=\"en\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['A',\n",
              " ' person',\n",
              " ' on',\n",
              " ' a',\n",
              " ' horse',\n",
              " ' jumps',\n",
              " ' over',\n",
              " ' a',\n",
              " ' broken',\n",
              " ' down',\n",
              " ' airplane']"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sim_matrix, sent1_toks, sent2_toks = aligner.get_sim_matrix(candidate=\"A person on a horse jumps over a broken down airplane\",\n",
        "                  reference=\"A person is at a diner ordering an omelette\")\n",
        "sent2_toks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAEcCAYAAABDIuaWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkSklEQVR4nO3de5RcVZn38e+vGxAI1wGJgQCJGAhBQpAQ5Y4oCC4URW4RZgZkgBm5qLzwDo7zAqKzAJkZHZVBowOIclFQ5BZBBwyXSEgCIQlBboJIEAVRuSkJSZ73j7MbKk2lu6prV9c53b/PWmelzq5TT+1S8mTvs/fZWxGBmZk1pqvTFTAzqxInTTOzJjhpmpk1wUnTzKwJTppmZk1w0jQza4KTpplZE5w0zcyasFqnK2DWLpJOrVP8AnBvRNw/yNWxIUJ+IsiGKklXAJOBG1LRgcACYAxwdUR8qUNVswpz0rQhS9IdwAcj4uV0vg5wE7A/RWtzQifrZ9Xke5o2lG0CLKk5fw0YGRF/7VVu1jDf07Sh7HLgHknXpfMPAVdIGgE82LlqWZW5e25DmqSdgV3T6cyImNvJ+lj1OWnakCapGxhJTa8qIn7TuRpZ1bl7bkOWpJOBs4DfA8sBAQFM7GS9rNrc0mwDSbtSTGupbd1c1rEKDVOSHgPeHRHPd7ouNnS4pZmZpO8CWwH3U7RuoGjdOGkOvqcoJrObZeOkmd9kYEK4CV8GjwMzJN1EzRSjiPjPzlXJqs5JM78HgLcBz3S6IsZv0rFGOsxa5nuamUn6OTAJmM3KrZsPd6pOZpaPW5r5nd3pCgx3kr4SEZ+WdAPF/eSV+B8wa4Vbmm0gaSSwczqdHRHPdrI+w42knSLiXkl71Xs/Im4f7DrZ0OGkmZmkw4ALgBkU8wL3AE6PiGs6Wa/hJk1qvywijux0XWxocdLMTNJ8YN+e1qWktwL/GxE7dLZm5ZaS3KKIGJ8x5l3APhGxNFdMM9/TzK+rV3f8ebyaVL8iYrmkhyVtkfExx8eBmZKuB16p+S5PObIBc9LM72ZJtwBXpvPDgekdrE+VbAgskjSblZPcQAdufpWOLmDd1qtn5u55W0g6GNg9nd4ZEdd2sj5V0a6BG0lrR8RfWolh1sNJM7O0VuOrqbu5DbAN8JOIeK3DVasESVsC4yLifyWtDXRHxEsDjLUL8D/AOhGxhaQdgBMi4pMZq2zDjO+15XcH8BZJmwE3A38LXNrRGlWEpOOAa4BvpqLNgB+3EPIrwAco7isTEfOBPVuIZ+ak2QZKXcGDgYsi4lBguw7XqSpOBHYDXgSIiEcptqwYsIh4qlfR8roXmjXISTM/pW7hkRSbeAF0d7A+VbKkdnqQpNWo80RPE55Ky/SFpNUlnQb8stVKDieStpT0/vR6LUnDfkDNSTO/TwGfBa6NiEWS3g78vMN1qorbJf0LsJakfYGreWP73YH4R4rW62bA0xRrApzYaiWHizq3S0bT2u2SIcEDQRmlCdrnR8Rpna5LFUnqAo4F9qN4muoW4NteZq8zJN0PTAHuiYgdU9nCiNi+oxXrMM/TzCiNmO/e/5VWT0SsAL6VjgGT9DX66NZHxCmtxB9GlkTEUklAltslQ4KTZn7z0hMoV7PyBO0fda5K1SBpN4pVorak+G9TQETE25sM1bPj5G7ABOD76fxQvHVvM3rfLvkkrd0uGRLcPc9M0iV1iiMiPjHolakYSQ8BnwHupWaUe6B7/EiaBeweEcvS+eoUDxu8J0N1h7x6t0sioqVewFDgpGmlIemeiHh3xngPA7tExB/T+YbArIjYJtd3DGWSPhUR/9Vf2XDjpJmZpK2Bi4CREfFOSROBD0fEFztctdKTdB7F9KwfsfKq9/cNMN4xFN39n1O0lPYEzo6I7wwg1tbA6bxx66CnbvsMpG7tImktYIuIeDhDrPsi4l29yub1DAoNV06amUm6neIv1zdrRhwfiIh3drZm5Ze2CuktBpKYUtfyPRQrHfW0Xu+JiN8NsG7zgW/w5lsH9w4kXjtI+hDw78AaETFW0iTgnGYXPJE0Ffg4xfoJd9a8tS6wIiLel6nKleSBoPzWjojZPSOOybJOVaaWpFt7/wdfr6xTIuK9GWOtkHRh+ofrugwhl0XERRnitNPZFFOEZgBExP2Sxg4gzi8oNgbcGPiPmvKXgAWtVbH6nDTz+4OkrUhTMyQdQod3ppS0JrA2sHG6r9eT0dejmPjdUZKOiojvSTq13vstrH95q6SPAT/KMNfzBkmfBK5l5VsHf2wxbk6vRcQLvf7Bbvp3R8STwJOS7ui9wpSk84F/bq2a1eakmd+JwDRgvKSngScoHqkcEEmHAjdHxEuS/hV4F/DFJu/znQB8GtiUonvZ87fqReDrA61bRiPSn7kf0TsBOBVYLunVVBYRsd4AYv19+vP0mrIAmp0O1U6LJH0c6JY0DjiFotU4UPvy5gR5QJ2yYcX3NNskLRHXNdBlzWriLIiIiWnS/Bcp9h86cyCjzJJOjoivtVIfK6+0lN7nKKYIQfFE1RciYsmqP1U3zj9RzMncCnis5q11gZkRcVSG6laWk2ZmkjYCzqK4iR7AXRQ34wc613BeROwo6VxgYURc0coIpqR3Ukz4XrOnLCIuG0isXPEkfbWv91t5gkfSh3ljObgZEXFjC7Fy/29XW7fbI6KlieOSJlMkzTG80YuMiJjYZJz1KVbRPxc4o+atl0p2O6IjnDQzk/QzijU1v5eKjgT2joj3DzDejRSLTexL0TX/K8W2wE1v1CbpLGBvir/40ym6WndFxCEDrFuWeJJ6ur51n+CJiH8cYP3Oo9hK+fJUNBWYGxGfHUCs3P/bnUsxaFNbtzkR8S8DiZdiPgycBjwArOgpT/coBxpzd4pFoS+RtDGwbkQ8MdB4Q0JE+Kg5KFqIF7bw+QfqlC1sId7aFGtzjkvno4D9BhhrIcXKVvPT+UjgZy3ULXe8WcBqNeerU0xGH2i8BRS3SHrOu4EFJfmt2epWE+OuVj5fJ95ZFI9NPpLON6Xonmf7jioeHggCJO1IMS/tUIqBm1aeE/+ppCOAH6TzQyjuLTVbp/Ui4kWKruCMVPY3FCO3c/v4aF/+GsVUnGWS1gOeBTYfYKx2xNuQYkS/pwu4TiprxQY18dZvIU7u35qzbj3OkvRt4FZWHuEf6H/PHwV2BO5LcX7r9TSH8eh5esJjajr+QNElVLQ+V/A4ipHq76bzbuAVSSfQ3MjtFcCBFKPdwRsj3jDwUdu5kjagWEXoXuBl4O4BxGlXvPMoFjxZ6QmeFuKdWyfeGX1/ZJVy/9acdetxDDCeooXe0z0PBt4IWBoRIaln+tyI/j4wHAzbe5qSVlA87XBsRDyWyh6P5lfUqSRJY4D1IiLLZOVc8SS9jQxP8NTEG0VxXxOKe8EtxUsxx5Dnt2atm6SHI+Nz9SpWuh9HcT/9XOATwBUxzGdgDOek+RHgCIrBh5uBqygWvB3IExRmHZdW2LogIrItf5eWhKtd5ehnuWJX1bBNmj1Sl+Mgim76PsBlFFtV/LSjFTNrkqRfUsytfILinmbPeqRNTTmyvg37pFkrPWJ4KHB4lOR5bLNGqdgz/k2iySlHkl6i/uOXPUl4IE9UDRlOmmZmTfBulGZmTXDSNDNrgpOmmVkTnDTNzJowbJ4IOmHrt2Qd8Tpj9zVyhmPZayv6v6gJbxvb6tOHb5g7q+X54CvZ+/zMG3Pe28qSkW/26IzHs8UaO2V0tlgAqx3ymazxmHtV1nD68O3q/6pVa+bv6TcfWdLSdw3UsEmaZlZ+Vej6OmmaWWmoI23H5jhpmllpuKVpZtaEKrQ0q5DY65L0EUkhaXyn62Jmeaymxo9OqWzSpFhg4670p5kNAWri6JRKJk1J61BsS3EsxfJuZjYEdCkaPjpWx459c2sOotgL/BHgeUk7dbpCZtY6tzTbZyrFosGkP91FNxsCutT40SmVGz1Pm4vtA2yf9i7pBkLS6dFrnTtJxwPHA+yxSTfbrt896PU1s8ZVoRVXhTr2dgjw3YjYMiLGRMTmFCtV79H7woiYFhGTI2KyE6ZZ+XWr8aNTqpg0pwLX9ir7Ie6im1We1PjRKZXrntfbYjcivtqJuphZXlVoxVUuaZrZ0NXJAZ5GOWmaWWlUIGc6aZpZeXRygKdRTppmVhq+p2lm1gTf0zQza0IFcubwSZr/vGvePX2+NXtJ1nhnfPStWeONGLVOtlg7TRmZLRYAK5bljbf12Kzhxrz4arZYq229ZbZYAFp7k6zxYsvtssZrlVuaZmZNEJ1bvahRTppmVhqdXFy4UU6aZlYaVdjuwknTzErDU47MzJpQhZZmKRK7JCdvM6OriaMRkvaX9LCkxySdUef9LSXdKmmBpBmSRjdSxywkvSzpy5IWpUq8NZVvJelmSfdKurNn90hJl0r6hqR7gC9J2kvS/emYJ2ldFS6Q9ICkhZIOT5/dO/3AayQ9JOlyqQr/RplZX3Ku3C6pG7gQOACYAEyVNKHXZf8OXBYRE4FzgHP7rWOzP6oPI4C5EbEdcDtwViqfBpwcETsBpwH/XfOZ0cCuEXFqeu/EiJhEsaDwX4GDgUnADsD7gQskjUqf3RH4NMX/GG8Hdsv4W8ysAzIvQjwFeCwiHo+IpRRb4xzU65oJwG3p9c/rvP8mOZPmCuD76fX3gN3TrpG7AldLuh/4JjCq5jNXR8Ty9Hom8J+STgE2iIhlFDtOXhkRyyPi9xTJeOd0/eyIWBwRK4D7gTEZf4uZdUDmRYg3A56qOV+cymrNp2icAXwUWFfSRn0Fbec9zUjx/xwRk2qObWuueeX1iyPOA/4BWAuY2dON70PtIznLqTOoJel4SXMlzb3ykdcG/EPMbHA0c0+z9u93Oo4fwFeeBuwlaR6wF/A0RT7ps465dFHs3wPwceCuiHgReELSoQDpHuUO9T4saauIWBgR5wNzgPHAncDhkrrTPdI9gdmNVqh2j6CpW68+8F9mZoOimXuatX+/0zGtV7ingc1rzkenstdFxG8j4uCI2BH4XCr7c591bP1nvu4VYIqkByh2izwnlR8JHCtpPrCIVd8z+HQa8FkAvAb8hGIvoAUUTejbgP8bEb/LWGczK5HM3fM5wDhJYyWtARwBXL/y92ljST158LPAxf0FzTrVJw3o9C57Ati/TvnRvc5PXkXY09NRe+0MYEbN+UlNV9bMSifnIsQRsUzSScAtFFt9XxwRiySdQzFofT2wN3Bu2g78DuDE/uJ6fqSZlUaX8i7YERHTgem9ys6seX0NcE0zMbMlzYjItxaZmQ1LXhrOzKwJVXhExUnTzEqjFM9198NJ08xKwy1NM7MmrFaBpuawSZpjD9w+a7x/OzjzZPnn/5Q13NI/vJwt1ohR62aLBcDIus83DNzTd2cNt/r4LfIF22hU/9c0Y80N88bbaJu88VpUgYbm8EmaZlZ+Hj03M2uC72mamTXBLU0zsyZUYBzISdPMyqO7AlnTSdPMSqMK9zRLn9clTU37Az0g6fya8pcl/Zuk+ZJmSRrZyXqaWetyb6zWDqVOmpI2Bc6nWJ9zErCzpI+kt0cAsyJiB4olnY7rRB3NLJ8uRcNHx+rYsW9uzM7AjIh4Lu0ZdDnF6u0AS4Eb0+t78R5BZpWXeRHitih70uzLaxHR889Nv3sETfvZ7we3dmbWtMy7UbZF2ZPmbIpNjzZOexhPpdiRsiG1e4gcv69veZqVXc59z9ul1KPnEfGMpDMo9iMWcFNEXNfhaplZm3hyewYRcSVwZZ3ydWpeN71kvZmVTxWmHJU+aZrZ8FH2+4XgpGlmJeKWpplZE7o6OSzeICdNMysNVaCp6aRpZqVRgZzppGlm5eGWZon88ur7ssYb+668e7+sWLo8a7xYke/Z3Btv+E22WAAH7/VE1ni8mHd/pWdunJct1qj3/jVbLAAmvJo33h8fzRtvdGsfd9I0M2tCBXKmk6aZlUdXBR4JctI0s9KQk6aZWePcPTcza4IHgszMmlCBnFndpCnpIoqV3dcCromIszpcJTNrUVdX+ZfsKH8NV+1zETEZmEixUPHETlfIzFqTe7sLSftLeljSY2lt3t7vbyHp55LmSVog6YP9xaxy0jxM0n3APGA7YEKH62NmLZLU8NFArG7gQuAAivwwVVLvPPGvwA8iYkfgCOC/+4tbyaQpaSxwGvC+iJgI3ASsWee61/cI+sGv8j5xY2b5qavxowFTgMci4vGIWApcBRzU65oA1kuv1wd+21/QSiZNih/5CvBC2u/8gHoX1e4RdNhW3YNaQTNrXs6WJrAZ8FTN+eJUVuts4ChJi4HpwMn9Ba1k0oyI+RTd8oeAK4CZna2RmeXQzD3N2p5kOo4fwFdOBS6NiNHAB4HvSn23Yys7eh4RR3e6DmaWVzOj5xExDZjWxyVPA5vXnI9OZbWOBfZP8e6WtCawMfDsKuvYcA3NzNos8+j5HGCcpLGS1qAY6Lm+1zW/Ad5XfLe2pRgbea6voJVtaZrZ0JPziaCIWCbpJOAWoBu4OCIWSToHmBsR1wP/B/iWpM9QDAodHRF9rqvopGlmpdHgqHjDImI6xQBPbdmZNa8fBHZrJqaTppmVhp89NzNrQgVy5vBJmtt+6Qt5Ay5fmjfemhvmjffaK9lCHXzA/GyxABg1JWs4bfHerPFGbfeRfMEi80MVIyZlDact8v13kkMVnj0fNknTzMrPLU0zsyb4nqaZWRMqkDOdNM2sPLxHkJlZE5w0AUkvR8Q67f4eMxsCnDRbI2m1iFjW6XqY2eCowkDQoEyKkvRvkuZLmpXWv0TSGEm3pSXmb5W0RSq/VNI3JN0DfEnSXpLuT8c8Seum606XNCd9/vOD8TvMrL1yb3fRDoORNEcAsyJiB+AO4LhU/jXgO2nl9cuBr9Z8ZjSwa0ScSrFC+4kRMQnYA/irpP2AcRQrM08CdpK05yD8FjNrpwpkzcFImkuBG9Pre4Ex6fUuFAsIA3wX2L3mM1dHvP4oxUzgPyWdAmyQuuv7pWMecB8wniKJrqR2kdJpV8zK94vMrC3UpYaPThmMe5qv1Sy1tLzB73z92a6IOE/STRSrKs+U9AFAwLkR8c2+gtQuUhpPXtDnck9m1nlVGD3v5IOev6BYFBTgSODOehdJ2ioiFkbE+RSLio6nWB/vE5LWSddsJmmTQaizmbVTVxNHh3Ry9Pxk4BJJp1OslHzMKq77tKT3AiuARcBPImJJWmX57jTa9jJwFH0sUW9m5VeF0fO2J83aOZoRcQ1wTXr9JLBPneuP7nVed3e4iPgv4L9y1tXMOstJ08ysGeXPmU6aZlYe6vZ6mmZmDatA79xJ08xKpAJTjpw0zaw0PBBUJvddlzfeqFF54/3u93njbbRBvli/ey5fLICNn8gaLp78RdZ4vPKXfLHeOjJfLICNt88aLp5dkDWeNji0tc+XP2cOo6RpZuVXgazppGlmpaFuJ00zs4b5nqaZWTMqMHretpmkkqZL2qBd8c1s6KnAcprta2lGxAfbFdvMhqgKdM/b2dL8taSN07YWD9SUnybp7PR6hqQvp4WCfylpZ0k/kvSopC+ma8ZIekjS5emaayStnd47T9KDacuLf2/XbzGzwSGp4aNTynBPc2lETJb0KeA6YCfgj8CvJH05XbMNcGxEzJR0MfBJSZcAHwXGR0T4VoBZ9VVh9LwMT8dfn/5cCCyKiGciYgnwOLB5eu+piJiZXn+PYmuMF4BXgf+RdDCQcUaymXVCFVqag5E0l/X6njV7vb8k/bmi5nXPeU9LuPdWFZH2CppCsT7ngcDNvb94pT2Cfvq7AVbfzAZNlxo/OmQwuue/BzaRtBHFCut1E1w/tpC0S0TcDXwcuCttdbF2REyXNJOiZbqSlfYIunZ37xFkVnbDeSAoiYh4DTgHmA38DHhoAHEeBk6U9EtgQ+AiYF3gRkkLgLuAU/NU2cw6JvOcI0n7S3pY0mOSzqjz/pcl3Z+ORyT9ub+YbWlpSuqmSGovAkTEV1l5X3NS+d41r2cAM3q/J2kMsCwijur18b9QdM/NbKjIuAhxykMXAvsCi4E5kq6PiAd7romIz9RcfzKwY39x29XSXAR8O7Uyzcwao67Gj/5NAR6LiMcjYilwFXBQH9dPBa7sL2hbWpoRMT5jrF8D78wVz8xKLO8Az2bAUzXni4F317tQ0pbAWOC2/oKWYcqRmVmhiXuatbNj0nF8C998BHBNRCzv78IyTG43Mys0MXpeOztmFZ7mjbneAKNTWT1HACc28r1uaZpZeeSdpzkHGCdprKQ1KBLj9b0vkjSeYlbO3Y0EdUvTzMqjuztbqIhYJukk4BagG7g4IhZJOgeYGxE9CfQI4KqIaGgu9/BJmuuvlzfeelvmjZfbX17JFyv3XtTrj80b7y1z88ZbsjRfrDX/Jl8sgBUZ6waw3hZ547Uq8+T2iJgOTO9Vdmav87ObiTl8kqaZlV8Fnghy0jSz8qjAyu1OmmZWHm5pmpk1obEnfTrKSdPMyiP3oGMbDKmkKam7kRn9ZlZSFeiedyytS5oqaaGkBySdn8r+UdIFNdccLenr6fVRkmanJZy+mVYwQdLLkv5D0nxgl478GDPLowKLEHckaUraFDgf2AeYBOws6SPADyn2/elxOHCVpG3T690iYhKwHDgyXTMCuCcidoiIuwblB5hZe+Rd5agtOtU93xmYERHPAUi6HNgzIn4s6XFJ7wEeBcYDMymeCd2JYj08gLWAZ1Os5RTJ1syqzt3zAbkKOAz4GHBterRJwHciYlI6tqmZxf/qqu5jrrRH0I2/GZTKm1kLursaPzqkU988G9gr7YveTbH45+3pvWspFgqdSpFAAW4FDpG0CYCkv0nr3/UpIqZFxOSImHz8gSV7XMzM3izzdhft0JHueUQ8k/br+DlFK/KmiLguvfentBfQhIiYncoelPSvwE8ldQGvUXTZn+xE/c2sTbrK2PldWcemHEXElaxiafmIOLBO2feB79cpXyd/7cysIypwT3NIzdM0s4pz0jQza4IX7DAza0JXvkWI28VJ08zKwy1NM7MmeJUjM7MmOGmWxx3n35I13i4feCRrvFeeeSlrvNVGrJ4t1nU//X22WABHbrpp1nivzVqYNd7dtzyRLdbOe+b972StvxuXNR6zfpA33qHHtfZ5+Z6mmVkT3NI0M2tcV/lTUvlraGbDh+9pmpk1wUnTzKwJTppmZk1w0mwfSRdRrAC/FnBNRJzV4SqZWas8ENRWn4uIP6ZFjG+VNDEiFnS6UmbWggq0NMtfw1U7TNJ9wDxgO2BCh+tjZq2qwMZqlUyaksYCpwHvi4iJwE3AmnWue32PoBsWx2BX08ya5aTZNusBrwAvSBoJHFDvoto9gj40uvyrp5gNexVImpW8pxkR8yXNAx4CnqLY5tfMqq4C9zQrmTQBIuLoTtfBzDLryrfQTLuUP62b2fCRuXsuaX9JD0t6LO2AW++awyQ9KGmRpCv6i1nZlqaZDUEZu+dpOuKFwL7AYmCOpOsj4sGaa8YBnwV2S9uHb9JfXLc0zaw88rY0pwCPRcTjEbEUuAo4qNc1xwEXRsSfACLi2f6COmmaWXnkTZqbUQwU91icymptDWwtaaakWZL27y+ou+dmVh5NPEYp6Xjg+JqiaRExrclvXA0YB+wNjAbukLR9RPy5rw8MC3t++TN5A27w9rzhXv1T1ngsfTlbqCM/lHe7C8a8P2u41Td9T9Z4e+4xP1+wkTvkiwVok3dljRf7jMwar2VN3NNMCbKvJPk0sHnN+ehUVmsxcE9EvAY8IekRiiQ6Z1VB3T03s/LI2z2fA4yTNFbSGsARwPW9rvkxRSsTSRtTdNcf7yvosGlpmlkFZBw9j4hlkk4CbgG6gYsjYpGkc4C5EXF9em8/SQ8Cy4HTI+L5vuI6aZpZeWR+IigipgPTe5WdWfM6gFPT0RAnTTMrDz9GaWbWBJU/JWVL65JaGq6V9Ot0I9bMhiup8aNDBjWtS1otIpYN5neaWYVUoHuetYaSvpweer9V0ltT2QxJX5E0F/iUpPdJmidpoaSLJb2lV4y1JP1E0nGSRqRrZqfPHJSuOVrSjyTdLOlRSV/K+TvMrFPUxNEZOZPmCIph/O2A24Hajc7WiIjJFA/PXwocHhHbU7R0/6nmunWAG4ArI+JbwOeA2yJiCvBe4AJJI9K1k4DDge2BwyXVTmI1syqqQPc8Z9JcAXw/vf4esHvNez3l2wBPRMQj6fw7wJ41110HXBIRl6Xz/YAzJN0PzKDY0mKL9N6tEfFCRLwKPAhs2btCtdtdTPuB91wzK71hvnJ77aY8rzT4mZnA/pKuSPOnBHwsIh6uvUjSu4ElNUXLqfNbah+zigdP8yZBZmU3zO5pdgGHpNcfB+6qc83DwBhJ70jnf0vRle9xJvAnim48FLP1T5aKtrikHTPW18xKp6uJozNyfvMrwBRJDwD7AOf0viB1pY8Brpa0kKJL/41el30KWCsN7nwBWB1YIGlROjezoaoC9zSzdc8jYp1VlO/d6/xW4E0txogYU3N6TM3rE+pceynFgFLP+YHN1NXMSqoC3fPyT783s2Gk/FttO2maWXl0dXe6Bv1y0jSzEnFL08yscR0c4GmUk6aZlYY8EFQiizM/EbTs1bzx/pJ5H56XXswX6w9/zBcLYGTm6bbPLswb75EH+7+mUUsy/v8AsP7YvPGezbgfEsBGH28xgFuaZmaNc/fczKwJ8ui5mVnj3NI0M2tG+QeCBr2GaQHhrw/295pZBQynZ8/NzFpWge75oLQ0JR0j6RFJs4HdasrHSLpN0oK0RcYWkrolPaHCBpKWS9ozXX+HpHGSzk7bYMyQ9LikUwbjd5hZuw2vpeHqkjQK+DxFstwdmFDz9teA70TEROBy4KsRsZxi3c0J6fr7gD3SXkKbR8Sj6bPjgQ8AU4CzJK3e7t9iZm1WgZXbB+Ob3w3MiIjnImIpb2x9AbALcEV6/V3e2CLjToptMPYEzk3lOwNzaj57U0QsiYg/AM8CI9v3E8xsUDhpDtgdwB4UrcjpwAbA3hTJtEe/212stEfQ9MVtq6yZZVKBgaDBSJr3AHtJ2ih1oQ+tee8XwBHp9ZG8kRRnA7sCK9Jq7/dTLEZ8RzNfHBHTImJyREw+/oOjW/gJZjY4htcWvnVFxDPA2cDdFBun/bLm7ZOBYyQtoNgv6FPpM0uAp4BZ6bo7gXWBzA8Zm1mpVKB7PihTjiLiEuCSOuVPUuwnVO8ze9S8voI37n0SEWf3uvaduepqZp1U1juGb/A8TTMrDy8NZ2bWjPJPbnfSNLMScdI0M2tC+bvn5a+hmQ0fmedpStpf0sOSHpN0Rp33j5b0nKT70/EP/cV0S9PMSiRfO05SN3AhsC+wGJgj6fqI6L2fyfcj4qRG4w6fpPnbzHvwrL123nhLl+aNt2x5tlDxwsvZYgFo7tVZ4/HCS1nDLX/pL9lidZP3SbQY83TWeNx/W95427YaIGvndwrwWEQ8DiDpKuAgoKVNoNw9N7PyaKJ7XvuYdDqO7xVtM4qHZHosTmW9fSyttHaNpM37q+LwaWmaWQU0PnoeEdOAaS1+4Q3AlRGxRNIJwHdYxQM3PdzSNLMSybqe5tNAbctxdCp7XUQ8nx7bBvg2sFMjNTQzK4msC3bMAcZJGitpDYrFga5f6duK9X57fJiV18aoK0vSlPRtSRP6v3Klz5wt6bQc329mQ0TGBTsiYhlwEnALRTL8QUQsknSOpA+ny06RtEjSfOAU4Oj+4ma5pxkRdec2SepOK7GbmTUgb+c3IqZTrMlbW3ZmzevPAp9tJmZTNZR0URqlWiTp8zXlMyRNTq9flvQfKXPvIunXkr4kaaGk2ZLeUSfucZLmSJov6YeS1k7ll0r6qqRfpL2ADqn5zOnpMwtq62JmVTb01tP8XERMBiZSLCw8sc41I4B7ImKHiLgrlb0QEdsDXwe+UuczP4qInSNiB4pm9LE1742i2O7iQOA8AEn7AeMo5mFNAnbq2XzNzCpsCK7cfpik+4B5wHasvElaj+XAD3uVXVnz5y51PvNOSXdKWkixgvt2Ne/9OCJWpFn8PfsA7ZeOeRQbr42nSKJmVmlDaDdKSWOB04D3pd0jbwLWrHPpq3XuY8YqXve4FDgptUY/3ytu7V5Aqvnz3IiYlI53RMT/1KnzG3sEzXi+r59nZqUwhJImsB7wCvCCpJHAAU189vCaP++u8/66wDNpD6EjG4h3C/AJSesASNpM0ia9L1ppj6C9N2qiumbWERXonjc8eh4R8yXNAx6ieDRpZhPfs2HaB2gJMLXO+/+PYgO259Kf6/ZTl59K2ha4W8X/eC8DR1Fs5WtmlVX+qeOKqNdbzvgF0q+ByWl/8o6JS3fI+0PfMTZruOwLdizJFy+ezLtIhN721qzxSr1gx4Z9/vvfvL3+Pm+8Oy/LGk5T57TWBFzxq8b/nnZt1ZHmpp89N7MS8crtRMSYdn+HmQ0R3ljNzKwZTppmZk1w99zMrHHunpuZNaP8Lc22TzkyMxtKyt8WNjMrESdNM7MmOGmamTXBSdPMrAlOmmZmTXDSNDNrwv8HAKebJv2XmboAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.tick_params(labelbottom = False, bottom=False, top=False, labeltop=True)\n",
        "plt.xticks(rotation=0)\n",
        "plt.yticks(rotation=90)\n",
        "\n",
        "ax = sns.heatmap(\n",
        "    sim_matrix.detach().cpu().numpy(),\n",
        "    xticklabels=sent1_toks,\n",
        "    yticklabels=sent2_toks,\n",
        "    cmap=\"YlOrBr\",\n",
        "    square=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>__id__</th>\n",
              "      <th>alignments</th>\n",
              "      <th>hypothesis</th>\n",
              "      <th>index</th>\n",
              "      <th>is_annotated</th>\n",
              "      <th>label</th>\n",
              "      <th>premise</th>\n",
              "      <th>reason</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>l3NpgcYFk70AfYKwwUsr</td>\n",
              "      <td>[{\"align_h\":{\"end\":5,\"start\":2,\"text\":\"is trai...</td>\n",
              "      <td>A person is training his horse for a competition.</td>\n",
              "      <td>0</td>\n",
              "      <td>True</td>\n",
              "      <td>Neutral</td>\n",
              "      <td>A person on a horse jumps over a broken down a...</td>\n",
              "      <td>the person is not necessarily training his horse</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>wrJnJZgf2cFrGxyBhO5Q</td>\n",
              "      <td>[{\"align_h\":{\"end\":1,\"start\":0,\"text\":\"A perso...</td>\n",
              "      <td>A person is at a diner, ordering an omelette.</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>Contradiction</td>\n",
              "      <td>A person on a horse jumps over a broken down a...</td>\n",
              "      <td>One cannot be on a jumping horse cannot be a d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>AS1Y7F6PkuKDVUn8xK6M</td>\n",
              "      <td>[{\"align_h\":{\"end\":3,\"start\":2,\"text\":\"is outd...</td>\n",
              "      <td>A person is outdoors, on a horse.</td>\n",
              "      <td>2</td>\n",
              "      <td>True</td>\n",
              "      <td>Entailment</td>\n",
              "      <td>A person on a horse jumps over a broken down a...</td>\n",
              "      <td>a broken down airplane is outdoors</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>h4tKjIUuPPEj1Yru3Nfx</td>\n",
              "      <td>[{\"align_h\":{\"end\":0,\"start\":0,\"text\":\"They\"},...</td>\n",
              "      <td>They are smiling at their parents</td>\n",
              "      <td>3</td>\n",
              "      <td>True</td>\n",
              "      <td>Neutral</td>\n",
              "      <td>Children smiling and waving at camera</td>\n",
              "      <td>Just because they are smiling and waving at a ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>kB16qtNITlEu86GayZvI</td>\n",
              "      <td>[{\"align_h\":{\"end\":3,\"start\":3,\"text\":\"present...</td>\n",
              "      <td>There are children present</td>\n",
              "      <td>4</td>\n",
              "      <td>True</td>\n",
              "      <td>Entailment</td>\n",
              "      <td>Children smiling and waving at camera</td>\n",
              "      <td>The children must be present to see them smili...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                 __id__                                         alignments  \\\n",
              "0  l3NpgcYFk70AfYKwwUsr  [{\"align_h\":{\"end\":5,\"start\":2,\"text\":\"is trai...   \n",
              "1  wrJnJZgf2cFrGxyBhO5Q  [{\"align_h\":{\"end\":1,\"start\":0,\"text\":\"A perso...   \n",
              "2  AS1Y7F6PkuKDVUn8xK6M  [{\"align_h\":{\"end\":3,\"start\":2,\"text\":\"is outd...   \n",
              "3  h4tKjIUuPPEj1Yru3Nfx  [{\"align_h\":{\"end\":0,\"start\":0,\"text\":\"They\"},...   \n",
              "4  kB16qtNITlEu86GayZvI  [{\"align_h\":{\"end\":3,\"start\":3,\"text\":\"present...   \n",
              "\n",
              "                                          hypothesis  index  is_annotated  \\\n",
              "0  A person is training his horse for a competition.      0          True   \n",
              "1      A person is at a diner, ordering an omelette.      1          True   \n",
              "2                  A person is outdoors, on a horse.      2          True   \n",
              "3                  They are smiling at their parents      3          True   \n",
              "4                         There are children present      4          True   \n",
              "\n",
              "           label                                            premise  \\\n",
              "0        Neutral  A person on a horse jumps over a broken down a...   \n",
              "1  Contradiction  A person on a horse jumps over a broken down a...   \n",
              "2     Entailment  A person on a horse jumps over a broken down a...   \n",
              "3        Neutral              Children smiling and waving at camera   \n",
              "4     Entailment              Children smiling and waving at camera   \n",
              "\n",
              "                                              reason  \n",
              "0   the person is not necessarily training his horse  \n",
              "1  One cannot be on a jumping horse cannot be a d...  \n",
              "2                 a broken down airplane is outdoors  \n",
              "3  Just because they are smiling and waving at a ...  \n",
              "4  The children must be present to see them smili...  "
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "df = pd.read_csv(\"./snli_train.csv\")\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "examples = []\n",
        "\n",
        "label_map = {\n",
        "    \"Entailment\": \"absolutely true\",\n",
        "    \"Contradiction\": \"absolutely not true\",\n",
        "    \"Neutral\": \"may or may not be true\"\n",
        "}\n",
        "\n",
        "for i, row in df.iterrows():\n",
        "    premise = row[\"premise\"]\n",
        "    hypothesis = row[\"hypothesis\"]\n",
        "    label = row[\"label\"]\n",
        "    alignments = json.loads(row[\"alignments\"]) \n",
        "\n",
        "    align_pairs = \"\"\n",
        "    for alignment in alignments:\n",
        "        align_pairs += f'({alignment[\"align_p\"][\"text\"]}; {alignment[\"align_h\"][\"text\"]}), '\n",
        "    \n",
        "    prompt = f'Given a context \"{premise}\" the conclusion \"{hypothesis}\" is {label_map[label]}. List all phrases that support this statement:'\n",
        "\n",
        "    examples.append({\"prompt\": prompt, \"output\": align_pairs})\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'prompt': 'Given a context \"A person on a horse jumps over a broken down airplane.\" the conclusion \"A person is outdoors, on a horse.\" is absolutely true. List all phrases that support this statement:',\n",
              " 'output': '(on a horse jumps over a broken down airplane.; is outdoors,), (A person; A person), '}"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "examples[2]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "authorship_tag": "ABX9TyOEaIFD/pN7oilIWAljQC+2",
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "Paraphrase.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3.8.5 ('base')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "vscode": {
      "interpreter": {
        "hash": "2043299c89c8cd0b4d1a6f5cf4529bd58e6a4e0fe3181a25e0d328c821cdc5c5"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "03171cdbff0c40e2a9598466930bbe68": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f2553c3e2b2b4214a75e7c0092de897d",
            "placeholder": "​",
            "style": "IPY_MODEL_917b468d2b1c4bbab80a49ebe7d47224",
            "value": " 1668/1668 [02:42&lt;00:00, 10.60it/s]"
          }
        },
        "0354bc75a1dd4a7f95b455453fc8738d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_59a6614a0a854a01b8b2c85001648871",
            "placeholder": "​",
            "style": "IPY_MODEL_85d6dcad7b9c427dbb46f2c73955d1ff",
            "value": " 792k/792k [00:08&lt;00:00, 89.3kB/s]"
          }
        },
        "0fa3a0e366fa4474a758b99e1c9e3ca9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0fcc6ea53c354b62b979ecf4d6c1245c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "111991ccfc4b48e3b023de8ed9f09d48": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1126c244e0d24e939e92e055740f04f9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "120f5850f5884406986ab27d52072dc9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "100%"
          }
        },
        "1553dc027e51406e9ea132e25b1e4ddf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "1aa0462c01ba4c31944d1ed41670ed2a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1d3a9a01d6b64fdd9d8deafbeecf09ae": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a9a11a07b834b6e99f2971651bb8250": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1126c244e0d24e939e92e055740f04f9",
            "placeholder": "​",
            "style": "IPY_MODEL_6d6cbc0886d540ce83868319458c046e",
            "value": " 1.20k/1.20k [00:05&lt;00:00, 229B/s]"
          }
        },
        "3547d1312d694841bd2de6a7040ad99d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "Downloading: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d769ce10acd64ba5b461b5d541b965fd",
            "max": 791656,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5550f5993b2046eda7bd4a44646194b9",
            "value": 791656
          }
        },
        "3abffa9077cf43ccaae7e907d2a06ba3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "413a3bc19fbb45a09b16aee63315008e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "424bc4bc00c04353827ef6532f38e376": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_50fd06b167474d529a610cb25be68974",
            "placeholder": "​",
            "style": "IPY_MODEL_4b20a09bd3444e449f9a97c8d04da889",
            "value": " 242M/242M [00:04&lt;00:00, 57.3MB/s]"
          }
        },
        "4b20a09bd3444e449f9a97c8d04da889": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "50fd06b167474d529a610cb25be68974": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5154b87298ce4077bca8dde67e58de13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b75fccffbfd247bd82c9afff18d6964d",
              "IPY_MODEL_2a9a11a07b834b6e99f2971651bb8250"
            ],
            "layout": "IPY_MODEL_1d3a9a01d6b64fdd9d8deafbeecf09ae"
          }
        },
        "5550f5993b2046eda7bd4a44646194b9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "564cade8cd174b029cd3fb0309f11ea0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c8877bd13efc4a249a394fba17ac757e",
              "IPY_MODEL_72ec234f136941edbbedf1ab97596a37"
            ],
            "layout": "IPY_MODEL_d059ada38ccf422587bb5a2965770fb5"
          }
        },
        "57603540902c44bb9b4f5eb540e6e7f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6ec5c313ccc04f8eb20d7de548c38d81",
              "IPY_MODEL_03171cdbff0c40e2a9598466930bbe68"
            ],
            "layout": "IPY_MODEL_da887a8692fb49b3af53ccf3545ad9b2"
          }
        },
        "582f4b1e62de4ac18d8913fa64411682": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "Downloading: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a668a78ce8bb437db40b872e9e305e82",
            "max": 242136741,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_eee45c1d726741d291e9d8cf5c442df6",
            "value": 242136741
          }
        },
        "59a6614a0a854a01b8b2c85001648871": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6d6cbc0886d540ce83868319458c046e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6ec5c313ccc04f8eb20d7de548c38d81": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "info",
            "description": "Validating: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f32bd274651f461495219423240cc89b",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b6935f3a3e0f43688e24ad9ecb0f8744",
            "value": 1
          }
        },
        "728c5193752b4923817e6f3308c30408": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "100%"
          }
        },
        "72ec234f136941edbbedf1ab97596a37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_111991ccfc4b48e3b023de8ed9f09d48",
            "placeholder": "​",
            "style": "IPY_MODEL_0fcc6ea53c354b62b979ecf4d6c1245c",
            "value": " 1.20k/1.20k [00:16&lt;00:00, 71.7B/s]"
          }
        },
        "79ab9454c46046f897a1b548ede87e89": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "Epoch 1: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_992565b9af974b3fbd273a3816eb9e18",
            "max": 21393,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_413a3bc19fbb45a09b16aee63315008e",
            "value": 21393
          }
        },
        "8461ae940e3349bea83021aec3ac90bc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "858358c5ff854976953ae82885215fc9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "85d6dcad7b9c427dbb46f2c73955d1ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "881e88c71fc746c08c4b83b995835e37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_79ab9454c46046f897a1b548ede87e89",
              "IPY_MODEL_ec32bd8bd31a4f68898477d1be39c338"
            ],
            "layout": "IPY_MODEL_728c5193752b4923817e6f3308c30408"
          }
        },
        "8ed59c36715845aa9f6e60614c1b1e13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "917b468d2b1c4bbab80a49ebe7d47224": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "926d241d3c9d4580ad46b3e196566008": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "info",
            "description": "Validation sanity check: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_eb1fce035c5242ab91a80d6577972951",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ac91897e332e477bac4358fe62bf4c14",
            "value": 1
          }
        },
        "98957ff00f894359aea0000a006c463f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dd566ff884e748a8a809fd1acc9c6fd8",
            "placeholder": "​",
            "style": "IPY_MODEL_e0f6af553b464390be3aeba125d1bf30",
            "value": " 2/2 [00:00&lt;00:00,  2.80it/s]"
          }
        },
        "992565b9af974b3fbd273a3816eb9e18": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "99c13a5d0ae14ebc9368e5c0997da1e9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_582f4b1e62de4ac18d8913fa64411682",
              "IPY_MODEL_424bc4bc00c04353827ef6532f38e376"
            ],
            "layout": "IPY_MODEL_8461ae940e3349bea83021aec3ac90bc"
          }
        },
        "a668a78ce8bb437db40b872e9e305e82": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ac91897e332e477bac4358fe62bf4c14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "b6935f3a3e0f43688e24ad9ecb0f8744": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "b75fccffbfd247bd82c9afff18d6964d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "Downloading: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_da28babc5c7446bdbab89e5d3ec6049b",
            "max": 1197,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fc01faeb2e334433827fb0ecdbbc2b81",
            "value": 1197
          }
        },
        "b84293a0ae72438c8c92061698397698": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bc3fb47151ff48188e9dcf24b5650460": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cff1f564dfb74ebd9a63b3dac35573b8",
            "placeholder": "​",
            "style": "IPY_MODEL_8ed59c36715845aa9f6e60614c1b1e13",
            "value": " 892M/892M [00:14&lt;00:00, 62.4MB/s]"
          }
        },
        "bc9d1ad29fe24958a9bf966562fd3a8c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd6fab875ce24430b55d97948c70395e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c8877bd13efc4a249a394fba17ac757e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "Downloading: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b84293a0ae72438c8c92061698397698",
            "max": 1199,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1553dc027e51406e9ea132e25b1e4ddf",
            "value": 1199
          }
        },
        "ce2a004205f747c1baa6d977883b7854": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3547d1312d694841bd2de6a7040ad99d",
              "IPY_MODEL_0354bc75a1dd4a7f95b455453fc8738d"
            ],
            "layout": "IPY_MODEL_858358c5ff854976953ae82885215fc9"
          }
        },
        "cff1f564dfb74ebd9a63b3dac35573b8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d059ada38ccf422587bb5a2965770fb5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d769ce10acd64ba5b461b5d541b965fd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "da28babc5c7446bdbab89e5d3ec6049b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "da887a8692fb49b3af53ccf3545ad9b2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "100%"
          }
        },
        "db644836130d4af38dc37a876cae00d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "Downloading: 100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bc9d1ad29fe24958a9bf966562fd3a8c",
            "max": 891691430,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3abffa9077cf43ccaae7e907d2a06ba3",
            "value": 891691430
          }
        },
        "dd566ff884e748a8a809fd1acc9c6fd8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e0f6af553b464390be3aeba125d1bf30": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "eb1fce035c5242ab91a80d6577972951": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ec32bd8bd31a4f68898477d1be39c338": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bd6fab875ce24430b55d97948c70395e",
            "placeholder": "​",
            "style": "IPY_MODEL_0fa3a0e366fa4474a758b99e1c9e3ca9",
            "value": " 21393/21393 [1:47:52&lt;00:00,  3.31it/s, loss=0.154, v_num=0, val_loss=2.35]"
          }
        },
        "eee45c1d726741d291e9d8cf5c442df6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "f2553c3e2b2b4214a75e7c0092de897d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f32bd274651f461495219423240cc89b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fb267d475796426fb446b34563a7f452": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_db644836130d4af38dc37a876cae00d4",
              "IPY_MODEL_bc3fb47151ff48188e9dcf24b5650460"
            ],
            "layout": "IPY_MODEL_1aa0462c01ba4c31944d1ed41670ed2a"
          }
        },
        "fc01faeb2e334433827fb0ecdbbc2b81": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "feff9a619c3a41d7b2d70af0a9b7d88f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_926d241d3c9d4580ad46b3e196566008",
              "IPY_MODEL_98957ff00f894359aea0000a006c463f"
            ],
            "layout": "IPY_MODEL_120f5850f5884406986ab27d52072dc9"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}