tokenizers-captum.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hugging Face \"Tokenizers\"와 PyTorch \"Captum\" 라이브러리 사용기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "최근 자연어 처리 오픈소스 생태계에는 정말 다양한 기능을 수행할 수 있는 라이브러리가 우후죽순으로 등장하고 있습니다.\n",
    "\n",
    "**Transformers**로 유명한 **Hugging Face**에서는 Rust로 구현된 빠른 속도의 **Tokenizers** 라이브러리를 내놓았고,\n",
    "\n",
    "페이스북이 운영하는 **PyTorch** 진영에서는 모델의 해석을 돕기 위해 **Captum** 이라는 라이브러리를 공개하였습니다.\n",
    "\n",
    "두 라이브러리 모두 최신 라이브러리이니 만큼 실제 활용 예제가 많지 않지만, 앞으로 여러 모로 활용하기 좋을 라이브러리인 것 같아 **한국어 데이터**를 적용해 해당 라이브러리를 공부해볼 수 있는 작은 **튜토리얼 자료**를 만들어보았습니다. 많은 도움이 되셨으면 좋겠습니다 :)\n",
    "\n",
    "P.S 본 튜토리얼은 **Tokenizers**의 [**공식 예제**](https://github.com/huggingface/tokenizers/tree/master/bindings/python/examples)와 **Captum**의 [**공식 튜토리얼**](https://captum.ai/tutorials/IMDB_TorchText_Interpret)을 참조하여 만들어졌습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 데이터 추출 및 훈련 데이터 생성"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 먼저 **NSMC 코퍼스** 내 훈련 데이터셋을 이용해 **문장 데이터**와 **라벨 데이터**를 구축해줍니다.\n",
    "- 훈련 데이터셋 내 `nan`으로 찍히는 문장이 약 **5건** 존재하므로 해당 문장들을 제거해주기 위한 **조건문**을 걸어줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# NSMC 훈련용 코퍼스 내 문장 및 라벨 데이터 추출\n",
    "\n",
    "f_train = pd.read_csv('data/ratings_train.txt', sep='\\t')\n",
    "train_pair = [(row[1], row[2]) for _, row in f_train.iterrows() if type(row[1]) == str]\n",
    "\n",
    "train_data  = [pair[0] for pair in train_pair]\n",
    "train_label = [pair[1] for pair in train_pair]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "추출한 문장 데이터와 라벨 데이터를 일부 출력해 데이터가 잘 구축되었는지 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "문장: 아 더빙.. 진짜 짜증나네요 목소리\n",
      "라벨: 0\n",
      "\n",
      "문장: 흠...포스터보고 초딩영화줄....오버연기조차 가볍지 않구나\n",
      "라벨: 1\n",
      "\n",
      "문장: 너무재밓었다그래서보는것을추천한다\n",
      "라벨: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 추출된 문장 및 라벨 데이터 일부 확인\n",
    "\n",
    "for data, label in zip(train_data[:3], train_label[:3]):\n",
    "    print(f'문장: {data}\\n라벨: {label}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Tokenizers 라이브러리 활용한 토크나이저 학습"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 프로젝트 전반에 사용될 **변수 사전**을 정의해줍니다.\n",
    "- **CLI 환경**에서는 `argparse`, `fire` 등 목적에 맞는 라이브러리를 이용해주는 것이 좋지만,\n",
    "- **주피터 환경**에서의 실험이므로 **딕셔너리**를 활용해 간단하게 구현해주었습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 프로젝트 전반에 사용될 변수 사전 정의\n",
    "\n",
    "params = {\n",
    "    'batch_size': 64,\n",
    "    'num_epoch': 15,\n",
    "    'lr': 0.003,\n",
    "    'dropout': 0.5,\n",
    "    'min_frequency': 3,\n",
    "    'max_len': 20,\n",
    "    \n",
    "    'vocab_size': 20000,\n",
    "    'embed_dim': 100,\n",
    "    'hidden_dim': 256,\n",
    "    'filter_sizes': [2, 3, 4],\n",
    "    'num_filters': 100,\n",
    "    'output_dim': 1,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "앞서 추출한 문장 데이터들을 **새로운 텍스트 파일**에 저장해줍니다. 해당 텍스트 파일은 **Hugging Face 토크나이저**를 **학습**하는데 사용됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 추출한 문장 데이터 텍스트 파일로 저장\n",
    "\n",
    "with open('train_tokenizer.txt', 'w', encoding='utf-8') as f:\n",
    "    for line in train_data:\n",
    "        print(line, file=f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Hugging Face**의 `BertWordPieceTokenizer`를 학습시킬 때 우리는 다음을 **인자**로 넘겨줄 수 있습니다.\n",
    "    - `vocab_size`: 사전의 최대 크기\n",
    "    - `min_frequency`: 단어의 최소 발생 빈도\n",
    "    - `special_tokens`: 스페셜 토큰으로 취급할 토큰들의 리스트\n",
    "    - `wordpieces_prefix`: 토큰 앞에 붙을 접두사\n",
    "- 각 인자들을 상황에 맞게 채워준 후, 토크나이저를 **훈련**시킵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers import BertWordPieceTokenizer\n",
    "\n",
    "# BertWordPieceTokenizer 초기화\n",
    "\n",
    "tokenizer = BertWordPieceTokenizer()\n",
    "\n",
    "# 앞서 제작한 텍스트 파일 활용해 토크나이저 훈련\n",
    "\n",
    "trainer = tokenizer.train(\n",
    "    files=['train_tokenizer.txt'],\n",
    "    vocab_size=params['vocab_size'],\n",
    "    min_frequency=params['min_frequency'],\n",
    "    special_tokens=['[PAD]', '[SOS]', '[EOS]', '[UNK]'],\n",
    "    wordpieces_prefix=\"##\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "훈련시킨 토크나이저의 **파라미터** 목록은 다음 명령어로 확인이 가능합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model': 'BertWordPiece',\n",
       " 'add_special_tokens': True,\n",
       " 'unk_token': '[UNK]',\n",
       " 'sep_token': '[SEP]',\n",
       " 'cls_token': '[CLS]',\n",
       " 'clean_text': True,\n",
       " 'handle_chinese_chars': True,\n",
       " 'strip_accents': True,\n",
       " 'lowercase': True,\n",
       " 'wordpieces_prefix': '##'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 토크나이저 파라미터 확인\n",
    "\n",
    "tokenizer._parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이후 사용될 **스페셜 토큰**들의 id 값을 **변수로 저장**해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "# 패딩 토큰 인덱스 확인\n",
    "\n",
    "print(tokenizer.token_to_id('[PAD]'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 스페셜 토큰 변수화\n",
    "\n",
    "pad_idx = tokenizer.token_to_id('[PAD]')\n",
    "sos_idx = tokenizer.token_to_id('[SOS]')\n",
    "eos_idx = tokenizer.token_to_id('[EOS]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 토크나이저에 **패딩 옵션**을 부여해줍니다.\n",
    "- 이때 인자로는 **패딩 토큰의 아이디**, **실제 패딩 토큰** 그리고 **문장의 최대 길이**를 넘겨주어야 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 토크나이저에 대해 패딩 옵션 설정\n",
    "\n",
    "tokenizer.enable_padding(pad_id=pad_idx, pad_token='[PAD]', max_length=params['max_len'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rust로 구현된 `encode_batch` 함수를 이용해 **전체 데이터셋**에 대해 토크나이즈를 수행합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 'encode_batch' 함수 이용해 훈련 데이터셋에 대해 토크나이즈 수행\n",
    "\n",
    "encoded_data = tokenizer.encode_batch(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "훈련 데이터:\t149995 개\n",
      "훈련 라벨:\t149995 개\n",
      "인코딩 데이터:\t149995 개\n"
     ]
    }
   ],
   "source": [
    "# 데이터 개수 확인\n",
    "\n",
    "print(f'훈련 데이터:\\t{len(train_data)} 개')\n",
    "print(f'훈련 라벨:\\t{len(train_label)} 개')\n",
    "print(f'인코딩 데이터:\\t{len(encoded_data)} 개')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**패딩 옵션**을 적용한 후, 토큰화가 수행된 결과를 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "토큰: ['20여', '##년이', '흘러', '##두', '##기억에', '남는', '사랑에', '대해', '생각해', '볼수있는', '유쾌한', '영화', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']\n",
      "\n",
      "아이디: [16780, 2749, 3168, 1220, 14117, 2108, 4033, 2763, 2784, 6144, 3934, 847, 0, 0, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "# 토크나이저 초기 훈련 결과 확인\n",
    "\n",
    "print(f'토큰: {encoded_data[2020].tokens}\\n')\n",
    "print(f'아이디: {encoded_data[2020].ids}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 토크나이즈 결과 후처리"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 라이브러리를 통해 토큰화 작업은 모두 완료되었지만, **후처리 작업**을 따로 수행해주어야 합니다.\n",
    "    - **Hugging Face** 측에서는 후처리 작업을 따로 수행하는 `post_processor` 파이프가 파이프 라인에 포함되어 있다고 설명하였지만, \n",
    "    - 현재까지는 **BERT** 관련 후처리 파이프만 존재하는 듯 합니다.\n",
    "- 먼저, 전체 토큰 리스트에 문장의 시작을 알리는 **'SOS' 토큰**을 추가해줍니다.\n",
    "\n",
    "\n",
    "- 이후 **최대 길이**를 넘는 문장들을 모두 **최대 길이**에 맞게 슬라이스 연산을 통해 잘라줍니다.\n",
    "- 문장이 짧아 패딩 토큰이 포함된 문장은 첫 번째 패딩 토큰을 **'EOS'** 토큰으로 변경해줍니다.\n",
    "- 패딩 토큰이 포함되지 않은 문장은 시퀀스 말미에 **'EOS' 토큰**을 삽입해줍니다.\n",
    "\n",
    "_cf) **[SOS] 토큰**과 **[EOS] 토큰**은 **자연어 생성** 모델에서 주로 사용되는 토큰으로 단순 분류 모델에서는 사용해주지 않아도 무방한 토큰입니다. 본 튜토리얼에서 굳이 후처리 과정에 본 토큰들을 추가한 이유는 **범용적인 토큰화-후처리 과정**에 대한 이해를 돕기 위함입니다!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 후처리 함수 정의\n",
    "\n",
    "def postprocess(input_ids):\n",
    "    input_ids = [sos_idx] + input_ids\n",
    "    \n",
    "    # 문장 최대 길이까지 슬라이싱\n",
    "    input_ids = input_ids[:params['max_len']]\n",
    "\n",
    "    # 패딩 토큰이 포함된 문장이라면 원 문장 말미에 <EOS> 토큰 삽입 \n",
    "    if pad_idx in input_ids:\n",
    "        pad_start = input_ids.index(pad_idx)\n",
    "        input_ids[pad_start] = eos_idx\n",
    "\n",
    "    # 패딩 토큰이 포함되지 않은 문장이라면, 시퀀스 말미에 <EOS> 토큰 삽입\n",
    "    else:\n",
    "        input_ids[-1] = eos_idx\n",
    "    \n",
    "    return input_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 정의한 **후처리 함수**를 이용해 모든 시퀀스에 **후처리** 로직을 적용합니다.\n",
    "- 이후, 후처리가 이전 시퀀스들에 잘 적용되었는지 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 후처리 함수 이용해 토크나이즈 문장 후처리\n",
    "\n",
    "processed_data = [postprocess(data.ids) for data in encoded_data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "후처리 결과: [1, 16780, 2749, 3168, 1220, 14117, 2108, 4033, 2763, 2784, 6144, 3934, 847, 2, 0, 0, 0, 0, 0, 0]\n",
      "\n",
      "후처리 결과 디코딩: 20여 ##년이 흘러 ##두 ##기억에 남는 사랑에 대해 생각해 볼수있는 유쾌한 영화\n"
     ]
    }
   ],
   "source": [
    "# 후처리 결과 확인\n",
    "\n",
    "print(f'후처리 결과: {processed_data[2020]}\\n')\n",
    "print(f'후처리 결과 디코딩: {tokenizer.decode(processed_data[2020])}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 간단한 CNN 문장 분류기 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 이제 문장 분류에 사용할 간단한 **CNN** 기반의 텍스트 분류기를 정의합니다.\n",
    "- 간단한 모델이니 만큼, 별다른 설명은 달지 않습니다.\n",
    "- 모델에 대한 이해를 필요로하시는 분들은 **튜토리얼 하단**에 **읽을 거리**를 참조해주세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from torch.utils import data\n",
    "\n",
    "torch.manual_seed(32)\n",
    "torch.cuda.manual_seed(32)\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CNN 모델 정의\n",
    "\n",
    "class CNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        self.embedding = nn.Embedding(params['vocab_size'], params['embed_dim'], padding_idx=pad_idx)\n",
    "\n",
    "        self.convs = nn.ModuleList([nn.Conv2d(in_channels=1, \n",
    "                                              out_channels=params['num_filters'], \n",
    "                                              kernel_size=(fs, params['embed_dim'])) \n",
    "                                    for fs in params['filter_sizes']])\n",
    "        \n",
    "        self.fc = nn.Linear(len(params['filter_sizes']) * params['num_filters'], 1)\n",
    "        \n",
    "        self.dropout = nn.Dropout(params['dropout'])\n",
    "        \n",
    "    def forward(self, input_ids):\n",
    "        # input_ids    = [배치 사이즈, 문장 길이]\n",
    "\n",
    "        embedded = self.embedding(input_ids).unsqueeze(1)\n",
    "        # embedded     = [배치 사이즈, 채널 개수, 임베딩 차원]\n",
    "        \n",
    "        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]\n",
    "        # conved_n     = [배치 사이즈, 필터 개수, 문장 길이 - 필터 리스트[n] + 1]\n",
    "        \n",
    "        max_pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]\n",
    "        # max_pooled_n = [배치 사이즈, 필터 개수]\n",
    "\n",
    "        cat = self.dropout(torch.cat(max_pooled, dim = 1))\n",
    "        # cat          = [배치 사이즈, 필터 개수 x len(필터 리스트)]\n",
    "\n",
    "        return self.fc(cat)  # [배치 사이즈, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 데이터 텐서 변환"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 이제 앞서 구축한 문장 및 라벨 데이터를 `torch.Tensor` 형으로 변환해줍니다.\n",
    "- 변환 이후, `DataLoader`를 이용해 **batchify** 시켜줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 문장 및 라벨 데이터 torch Tensor로 변환\n",
    "\n",
    "processed_data = [torch.LongTensor(data).to(device) for data in processed_data]\n",
    "train_label = [torch.FloatTensor([label]).to(device) for label in train_label]\n",
    "\n",
    "\n",
    "# torch Tensor로 변환한 데이터 이용해 Iterator 정의\n",
    "\n",
    "train_iter = data.DataLoader(processed_data, batch_size=params['batch_size'])\n",
    "label_iter = data.DataLoader(train_label, batch_size=params['batch_size'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 데이터셋 이용해 CNN 모델 훈련"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 본격적으로 기정의한 모델과 데이터셋을 이용한 학습을 시작합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Train Loss: 0.473\n",
      "Epoch: 02 | Train Loss: 0.353\n",
      "Epoch: 03 | Train Loss: 0.313\n",
      "Epoch: 04 | Train Loss: 0.283\n",
      "Epoch: 05 | Train Loss: 0.252\n",
      "Epoch: 06 | Train Loss: 0.216\n",
      "Epoch: 07 | Train Loss: 0.192\n",
      "Epoch: 08 | Train Loss: 0.174\n",
      "Epoch: 09 | Train Loss: 0.148\n",
      "Epoch: 10 | Train Loss: 0.141\n",
      "Epoch: 11 | Train Loss: 0.129\n",
      "Epoch: 12 | Train Loss: 0.124\n",
      "Epoch: 13 | Train Loss: 0.113\n",
      "Epoch: 14 | Train Loss: 0.108\n",
      "Epoch: 15 | Train Loss: 0.101\n"
     ]
    }
   ],
   "source": [
    "model = CNN()\n",
    "model.to(device)\n",
    "\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "criterion.to(device)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=params['lr'])\n",
    "\n",
    "for epoch in range(params['num_epoch']):\n",
    "    model.train()\n",
    "    epoch_loss = 0\n",
    "    \n",
    "    for (batch, label) in zip(train_iter, label_iter):\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        logits = model(batch).squeeze(1)    # [배치 사이즈]\n",
    "        labels = label.view(label.size(0))  # [배치 사이즈]\n",
    "\n",
    "        loss = criterion(logits, labels)\n",
    "        epoch_loss += loss.item()\n",
    "        \n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "    train_loss = epoch_loss / len(train_iter)        \n",
    "    print(f'Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Captum을 이용한 모델 해석"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch 진영에서 공식적으로 관린하는 **Captum**은 모델 해석을 위한 **여러 기법**들을 **구현한 모듈**을 제공해주는 라이브러리입니다.\n",
    "\n",
    "이번 튜토리얼에서는 **Captum**을 이용해 간단하게 모델이 문장 내 어떠한 부분을 보고 판단을 결정하는지에 대해 살펴볼 예정입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 본 튜토리얼에 사용되는 **Captum**의 `LayerIntegratedGradients` 모듈이 현 배포판에 포함되어 있지 않다고 합니다. (2020년 1월 기준)\n",
    "- 때문에 [**Github repo**](https://github.com/pytorch/captum)를 clone해 수동으로 설치해주셔야 합니다!\n",
    "- 본인이 이용하는 파이썬 버전의 `site-packages` 폴더로 이동해 다음 명령어들을 실행시켜주시면 됩니다.\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/pytorch/captum.git\n",
    "cd captum\n",
    "pip install -e .\n",
    "pip install -e .[tutorials]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "설치를 마친 후, 본 튜토리얼에 사용되는 **API 모듈**을 임포트 및 정의합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 결과 해석에 필요한 Captum API 임포트 및 정의\n",
    "\n",
    "from captum.attr import LayerIntegratedGradients, TokenReferenceBase, visualization\n",
    "\n",
    "token_reference = TokenReferenceBase(reference_token_idx=pad_idx)  # 레퍼런스 생성을 위한 모듈\n",
    "lig = LayerIntegratedGradients(model, model.embedding)             # 결과 해석에 사용되는 IntegratedGradient 기법 모듈"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 이제 결과 분석을 위해 **비교 문장**을 생성할 `interpret_sentence` 함수와,\n",
    "- 각 결과를 `VisualziationDataRecord` **클래스 인스턴스**로 저장해 줄 `add_attributions_to_visualizer` 함수를 정의합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "vis_data_records_ig = []\n",
    "\n",
    "label_vocab  = {0: '부정', 1: '긍정'}\n",
    "\n",
    "def interpret_sentence(model, sentence, label = 0):       \n",
    "    model.zero_grad()\n",
    "    \n",
    "    input_ids = tokenizer.encode(sentence)\n",
    "    input_tokens = input_ids.tokens[:params['max_len']]\n",
    "    \n",
    "    input_ids = postprocess(input_ids.ids)\n",
    "    input_indices_tensor = torch.LongTensor(input_ids).to(device).unsqueeze(0)\n",
    "\n",
    "    # 단일 문장에 대한 예측 작업 수행\n",
    "    pred = torch.sigmoid(model(input_indices_tensor)).item()\n",
    "    pred_ind = round(pred)\n",
    "\n",
    "    # 베이스 라인 역할을 할 Reference 생성: 주로 패딩 토큰으로 채워줌\n",
    "    reference_indices = token_reference.generate_reference(params['max_len'], device=device).unsqueeze(0)\n",
    "\n",
    "    # LayerIntegratedGradients 모듈 활용해 개별 단어의 속성값 및 델타값 근사치 계산\n",
    "    attributions_ig, delta = lig.attribute(input_indices_tensor, reference_indices, n_steps=500, return_convergence_delta=True)\n",
    "\n",
    "    print('pred: ', label_vocab[pred_ind], '(', '%.2f'%pred, ')', ', delta: ', abs(delta))\n",
    "\n",
    "    add_attributions_to_visualizer(attributions_ig, input_tokens, pred, pred_ind, label, delta, vis_data_records_ig)\n",
    "    \n",
    "\n",
    "def add_attributions_to_visualizer(attributions, text, pred, pred_ind, label, delta, vis_data_records):\n",
    "    attributions = attributions.sum(dim=2).squeeze(0)\n",
    "    attributions = attributions / torch.norm(attributions)\n",
    "    attributions = attributions.cpu().detach().numpy()\n",
    "\n",
    "    # 시각화 위해 샘플을 리스트에 추가\n",
    "    vis_data_records.append(visualization.\n",
    "                                VisualizationDataRecord(\n",
    "                                    attributions,\n",
    "                                    pred,\n",
    "                                    label_vocab[pred_ind],\n",
    "                                    label_vocab[label],\n",
    "                                    label_vocab[1],\n",
    "                                    attributions.sum(),       \n",
    "                                    text,\n",
    "                                delta\n",
    "                                )\n",
    "                           )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 앞서 정의한 함수들을 활용해 기존에 학습되지 않은 **임의의 문장**을 실험 결과에 추가합니다.\n",
    "- 아래 문장들은 제가 최근 재밌게 보았던 영화 **Knives Out**의 네이버 리뷰입니다.\n",
    "- 튜토리얼이라는 목적에 맞게 모호한 리뷰보다는 **모델이 구분하기 쉬울만한 리뷰**들을 따로 선별하였습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pred:  긍정 ( 1.00 ) , delta:  tensor([0.9419], device='cuda:0')\n",
      "pred:  부정 ( 0.00 ) , delta:  tensor([11.6137], device='cuda:0')\n",
      "pred:  긍정 ( 1.00 ) , delta:  tensor([4.3984], device='cuda:0')\n",
      "pred:  부정 ( 0.00 ) , delta:  tensor([21.0811], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "# 예제 문장 추가 및 분석 수행\n",
    "\n",
    "interpret_sentence(model, '간만에 쫄깃한 추리 수사극을 볼수 있었습니다.명 배우들의 연기에 시간 가는 줄 몰랐네요.올해 본 영화중 가장 만족도 높은 영화였습니다.', label=1)\n",
    "interpret_sentence(model, '평이 워낙 좋아 갔는데 거품이 심한듯.. 개연성도 엉망에 르부아 블랑의 뜬금없는 직관에만 의존한 스토리 전개 ;; 평점 거품이 심한듯', label=0)\n",
    "interpret_sentence(model, '너무 재미있게 봤습니다. 클래식한 추리소설 느낌.타 인기영화때문에 개봉관이 적어 아쉽네요', label=1)\n",
    "interpret_sentence(model, '아, 2시간 넘게 지루하고 어이없는 추격전과 의욕이 넘쳐 과욕으로 변한 연기, 게다가 예상보다 너무 뻔한 결말로 고문당함.', label=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "앞선 함수의 호출로 총 4건의 결과가 저장된 `vis_data_records_ig` 리스트를 활용해 **시각화 결과**를 그려주도록 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table width: 100%><tr><th>Target Label</th><th>Predicted Label</th><th>Attribution Label</th><th>Attribution Score</th><th>Word Importance</th><tr><td><text style=\"padding-right:2em\"><b>긍정</b></text></td><td><text style=\"padding-right:2em\"><b>긍정 (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>긍정</b></text></td><td><text style=\"padding-right:2em\"><b>1.49</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 간만에                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 쫄기                    </font></mark><mark style=\"background-color: hsl(120, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ᆺ한                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 추리                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 수사                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##극을                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 볼수                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 있었습니다                    </font></mark><mark style=\"background-color: hsl(120, 75%, 52%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 명                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 배우들의                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 연기에                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 시간                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 가는                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 줄                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 몰랐네                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##요                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 올해                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 본                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>부정</b></text></td><td><text style=\"padding-right:2em\"><b>부정 (0.00)</b></text></td><td><text style=\"padding-right:2em\"><b>긍정</b></text></td><td><text style=\"padding-right:2em\"><b>-1.89</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 평이                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 워낙                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 좋아                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 갔는데                    </font></mark><mark style=\"background-color: hsl(0, 75%, 90%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 거품이                    </font></mark><mark style=\"background-color: hsl(0, 75%, 82%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 심한                    </font></mark><mark style=\"background-color: hsl(0, 75%, 89%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##듯                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 개연성도                    </font></mark><mark style=\"background-color: hsl(0, 75%, 72%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 엉망                    </font></mark><mark style=\"background-color: hsl(0, 75%, 89%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##에                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 르                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##부                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##아                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 블                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##랑                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##의                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 뜬금없는                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 직                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>긍정</b></text></td><td><text style=\"padding-right:2em\"><b>긍정 (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>긍정</b></text></td><td><text style=\"padding-right:2em\"><b>1.65</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 너무                    </font></mark><mark style=\"background-color: hsl(120, 75%, 89%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 재미있게                    </font></mark><mark style=\"background-color: hsl(120, 75%, 60%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 봤습니다                    </font></mark><mark style=\"background-color: hsl(120, 75%, 78%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 클래식                    </font></mark><mark style=\"background-color: hsl(120, 75%, 90%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##한                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 추리                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##소설                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 느낌                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 타                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 인기                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##영화                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##때문에                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 개봉                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##관이                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 적어                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 아쉽네요                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [PAD]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [PAD]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>부정</b></text></td><td><text style=\"padding-right:2em\"><b>부정 (0.00)</b></text></td><td><text style=\"padding-right:2em\"><b>긍정</b></text></td><td><text style=\"padding-right:2em\"><b>-2.27</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 아                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 2시간                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 넘게                    </font></mark><mark style=\"background-color: hsl(0, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 지루하고                    </font></mark><mark style=\"background-color: hsl(0, 75%, 74%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 어이없는                    </font></mark><mark style=\"background-color: hsl(0, 75%, 83%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 추격                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##전과                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 의                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##욕이                    </font></mark><mark style=\"background-color: hsl(0, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 넘쳐                    </font></mark><mark style=\"background-color: hsl(0, 75%, 87%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 과                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##욕                    </font></mark><mark style=\"background-color: hsl(0, 75%, 83%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##으로                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 변                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##한                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 연기                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 게다가                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> 예상                    </font></mark></td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 시각화 결과 표 변환\n",
    "\n",
    "visualization.visualize_text(vis_data_records_ig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래는 위 함수를 통해 반환된 **HTML 테이블**을 스크린샷으로 찍어 주피터 셀에 그려준 예입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABMYAAAE5CAIAAAD0k7vwAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAAGAmSURBVHhe7b3dy23Xedi7/oHeG/oHmLa04HMVnEBAtVMnKJL3a20OteldEgiySe0j53BQpa29vV0cXNIEnIBC5S1vaSvQgkuTi7pYlmJvS7owcSE3KhW2bO3WunLhmB43qRLnjK85xzOe8TE/11zzXe/vx4P0rrnGmh9jjjHH81tjzrUPfwsAAAAAAAAwC5QSAAAAAAAAZnL4/wAAAAAAAABmgVICAAAAAADATFBKAAAAAAAAmAlKCQAAAAAAADNBKQEAAAAAAGAmKCUAAAAAAADMBKUEAAAAAACAmaCUAAAAAAAAMBOUEgAAAAAAAGaCUgIAAAAAAMBMNlPK79+5OJR56pVQ5Bz5/p1Hw2EKbrwa3l3AKzfcmmzd/Ve3jYs73/fvrIzY0Il45Sm7B0sPMKmldQ7K79gaZ/M0ULEAAAAAsJyroJRm04/e+a/hxcYUldKwOF2epZSv3phe2ydXSr8DhsJJNO4ha1K9TFjLfEQdXnLzoWIBAAAAYAVOcOOrt6xHl02NjMbnuCdWSnmwPl1eLNKTc/cgt5dNKf1u33jK7UWy8+FLik481Ms2Mw9qbh3uESoWAAAAAFZhD0oZctZAN9vWFbvjktRQPqSe7qVez6u+oKVb6BPcjlMkrNnBdvvpDrN9jIY0j4+Hc+NVkbtns5TBWh1+DXKddjtOsCdv6AT4tmG23v8hl3dc/N7v6SYUdvtOOMYbrxQn02KbiV86qCmy/gzmdZhNpsV6kyuMJ6i0uRNBxQIAAADAOpxeKaX/eHKzMpj8Ui159EKsJ+aUAbdcZqJ7UcpwvG5nBo/R0CXW6bEEnAmkSpnXZ7Zam3bP2dD2+EMTddXV5Ejz6TBLCuajCMc403z8+lMab/Xn6zRQsQAAAACwEvu68TUkl4lu6fmT7oMheXUvw1tdturf8h/0f59s7iJNlyN+V7NjTPdWJuvBmUPJ5IOyWEiyuzX4T8n6dH/P3NDmpBLi9jkKgzrpxTbgm4dDHqN+N2laNfPp/+7qMCnp/+53L5RMTkq3zrSGTwIVCwAAAABrsRelDJmip6BABp+txnxRrCcksinel/xbu1LKuDP6GLsUOcUesjhYj6gNmdMLh1Qk25q3oa1pnFbDGPMR571gProt+cqZZT5+67KZiR0Itd1v7oRV6qFiAQAAAGA19nLjq080ZWapiuX5oijg31L4NFQkoKcgO4oE/e4805M5/TkpZZh0yghHN8Z8xG5PNJ++BuTLpA67t8aaT9h0/9YpqtRDxQIAAADAepxaKYPbhCzQp5I+s0yKWUJq2y3x6aN/qbJeSZ6Sbkp2FAnZuyJdVqQ39YUU3L+UiXWoz24N/lP+rcQ2Z21oW4RX9CTOoAqkLzO7KJhPX/NJ05KNUJdM6jDZYvhUpzeh3uRJ6d7q1nmCKvWoenNQsQAAAHAZ0aN/l7uKVEcnDPOopRl++clcYyecfJYyZJYJLrNMi1n6JuJp/zxP125CUmvpEtYtyY9CMniMljQFT3HNOu0nIQUXZFVkG/2cDW2JP6hsu/7o/BHFI3XHnr7Mun3JfFJC4ULNVOrQbzFcsMIOJyRvdSeo2/rmVeqhYgEAAOB8ULNHpcw//eZ6LrU0wy8/oVKaQz7h1gM7eJYyZpPmJMWzpYs5+qzULK+vR2aZ8SNy4WYUj6KnfYyWpPXHfH38PyIiVi4/bl9P3tCGhH3LO78/y/5gk5ZTbUiBgvnIf+siOUApUa+4z3XVmNRhYj6W+K79RN+392Q+VCwAAACcE2k6LXKGLkPI8op51NIMv/xUUnfarUdOoJRz6b51CNlweKl8DAAAAAAArgr+i2wvCOHvO2LqUk1jii+4pUd031Pf8e8m35Vb6rMsUuqCnvT/fnco33/V3n2n32nwK8Fu5J44xu3kk/+X+39A1kBHJ9Ldzvs1OBLHlp+SEw9iedu5LpFShtpPOb2UAwAAAADAiYiy189YxpnJ5J6mboJK4g1KuJbFLow+KRillA28yJWkpne28Tv5pWQPTRlVwOJ3uHksqYVa/Oay5Q2rvExKaZDKjk8CAAAAAFxt+nnI/o9Oh4wa9X8YgiN1BhEEzL3sZCzO3YXCwbs6CRynlKmS+XV6i/FWptaWvJy0k8nWFfKtoJTBCeU6u33udNGXNHsSlnfb6peXuWRKCQAAAAAA0NNNTt4R2uMUqPvtBjk3KKfaouYFy4rKlBWuOZVfnuhZ0LB0nXKFlZXblUzayXTrPX6hRyplX0y4d3irdlwKtaEISgkAAAAAAJeWbj7Q4mcIO7tzBBEatrX4m38165uulN06V1NKsZPp1tUW5Vtq51FKAAAAAACASPSfXsa8m1l6Bxu8p1Ta2oIbX0cqZVyb3NaknQxbDIXlvbXJB1tKGfa5q7f+cNJjGeJ0SlmolGF8TY08tsCsDe0dam91qNIjIb84vDpHDQAAANsR/KfTJEefgYR5S0NfTODfLaRnxWm6dZUyoZO6STspCj/1Smm1g0qZpmoev4na8hIo5eWE2lsdqvQYTLkYAQAAAMxDz/tZghOqPM0nb57O4mrpWbTKaf+IyDilNH/3OxP3xDF+J6NG2uXRMM2nxBabSmmoTQDUlmeglJcTam91qNIj4OsnXAr9gcvrFwAAAMDVQ+rlebBXpUwnN/qUvcvg43cG8mRETZdZ/llm8NTe6lClR8DXz6hajTelGGJ9JgraVV3yzxCLlZzTpRkAAADOFZRyPRqJtX8rIUzUhgw1xSfrMn33hCT+LDN4am91qNJjUJVGbekW+ZxASlJ1PaZ8thKsEgAAAHYOSrkeYxNrP18RZirUpEfI2u3LpFiy8rPM4Km91aFKj4NS6+7qGe7171766rKiXq/Srup6w+xW0r2MKwEAAACAzdizUvoE0ZNk8F0G2aeYN17pcs0Ul1yeZQZP7a0OVXpE4sPiBlddNf3LnhcPJd2SvnqTtxTyswAAAABwdHZ842t4SySUZPA91N7qUKVHR85M+mqcpZSx6vxbCvlZAAAAADg6e1RKn6aHO+JCap5k8P2nqvcZSs4yg6f2VocqXR9dCaImpV7ad/qS9SrNqy6sJCo9AAAAAGzOqZUyw6SYIYlMSDP4FJ9QFj5V/VdBLz/U3upQpUegVD+dYeY/zyO9MSVIY151tZUAAAAAwFbsUSn7yQf/0qfmbnnIUOW/2SAnKJIk3qfvhrPM4Km91aFKj0NSCabauhlLixTCpEJifSYfKVZddSUAAAAAsAWnU0oAAAAAAAC45KCUAAAAAAAAMBOUEgAAAAAAAGaCUgIAAAAAAMBMUEoAAAAAAACYCUoJAAAAAAAAM0EpAQAAAAAAYCYoJQAAAAAAAMwEpQQAAAAAAICZoJQAAAAAAAAwk8PfAgAAAAAAAMwCpQQAAAAAAICZoJQAAAAAAAAwE5QSAAAAAAAAZoJSAgAAAAAAwExQSgAAAAAAAJgJSgkAAAAAAAAzQSkBAAAAAABgJiglAMAl5sd//eOTxJt/9eZliVBTm/Pez947SfzgvR9clvjpz35KnCT+3g/+3sgIrRkAoAlKCQBwiVGmt1kobdtzhJraHGV6m4XStj2H8hxis1De2IjQmgEAmqCUAACXGGV6m4XStj1HqKnNUaa3WSht23MozyE2C+WNjQitGQCgCUoJAHCJUaa3WSht23OEmtocZXqbhdK2PYfyHGKzUN7YiNCaAQCaoJQAAMt4cO/i+r13w4shxhZ+9971w+03hssr01sr3nzx4YdffFMtlKG0baP41m8dPvbF+8nCP/6tw2/9cbJER6ipMq/fPlzcexBeTOLdly4uXmqdSWV6K8WDu9cPt15XC5NQ2rbnUJ6zy3j7zvXDjdfVwlpMKjwpJqz51ZuHR++9rRaqUN7YiNCax/DGbdM0w99zef3WwXWr7gJ4YuZfHzKmrGrkMGEq3BUzlbZSXU2o9sEL4AS6A7kymMZwe0RX6dqMaQ+jyp8YlBIAYAT2mq7oLvH58G8GSE2XTBRzhbjyPueYp5RvPvtYWFGZx559Mxb+2pNhaaTXSJTSYd7V9FnUQqW8f3PADF3cv3W4dvcduWQbpfzO71cb0hNf1YVtfOeFj4T3Oz7ywndUmTyU55h4+96jh5uvqoU//dmrNw6P3nlHLUzCeNQyl7PO1nHj1XR5tmazPz2DhWOYo2vKXuMw9ZptRWnCnhxVKY29aPoL1DSlNJc40fWEHbWUMr8OVy6PpnuGAhklZZI9XV4Q2teHIslFQ2xLr6pVk6XLfizfV/IipSzWsF5YqMZu65OVsjEsdgeSUmkDk7HraTBxE/W16UMoDB/dtsxbShGLLa1baJv9Jkqpz9G0jaKUAADTkUNgPvybd5PUSowWhVzBDjx+pHHjt7+Id6NpKbeQKNPz8bVnDodnviZftv3Qhyw2XSnvf/FjH/7i993f3//ih4X4/fGNw299y/89rH+lMJ8q8+G792coZSGTkzRz4i7ftSxTSuOKh8PN+9lyFeso5VefOTzxbf/3V5+QTvjtJw7PfDUrU4+3f/8jJaU0nz089vvfSRY6KdULdSjPMTFXKZ3jFT44MuzHo4a9bl71ophboi0cl9jC/b61ldK+e7h+5229vI+JSlk53mMrZdLsVQ56TKV0l0eddhcXGiYIj83XxbaSl8VEv45alU7Qm6tqjin2MMMSO16EQxOVpupqBIUariyMTLoADiHqVh57ZGBn5mEOwbBgtWm77RkaqdPDMcd+VKUUO5numGgqeh9czaRbcQ14fF2hlAAAk5Eja2EsMRfi8UqZFO5Hne6PoYFKmZ6NHzz7sB00H372B+7la34y8smvyTKFePPZx7qPzFFKoXbf+q3DjT/ulldVc6VYNks5cciclFEp0xNhtfBw85YZwYcmKldRSmN3H/n9t93fxgmF5n3nhY90c4miTCPKSmk0tfjZVF9LoTzHRGnyzdNQSqdqN28kpjcpjBampiesbNjlGoVF2LdMMfvxqlVeTqXsL1D6utcmTc279XTrl/l3wLxV7KrF5eOFRx+RIR5UMdGvkq9K7MbAqpId1pf9cl3JSht/Eeso1HBlYU9yCEuV0hxjLzCxwiXtnZmMqSWL2ZDdtGXWytNz0TM0UqeHo3Wu0jy6hbKuRiHWb+o2STC6ragdNsWKm6gtL4FSAgBMRF1k87EkuYgbxGiRFVYDc5eUdMPP0EClTM/75JOvmb/7+1qtTBpFHLBK+8FYYLJSCl0U05ImovLdv/thN6/YvzUpkrlKsZ4FSmnH6cDI3EKmjIMZlTK9EC6vuXbvgXvp3PJw674skISdzEwFcoZSRrUTDmkjzkymqlmNslIWJyS3mqW0k4SJ0aV3oo6KfKNiiXa5XNhsYUGulOYjyfJkn2WglNkFUGB63KRZSlvpJdRq9REZ4kEVE/0qZlX6ShLrpL2q9Hj1ZV8YgsFeuNxLUWkjr2ACu8XsClao9h6zFXl+By+AbewJ6tcWK1zS2pkJmJUHUjWqLR/A7lUZfQjmrGm6w+nf6puEaB5iYAoF+jM+EtF+0rYRG5I6fcnpSBA7NgRKCQ1coy/0c4CrjO0XyTgXB4Duoh/TCE9xtAiFVTYgMyG7XIwNRRLTs1oYZxqdVQqNtNOV8t0k1M2x45XSCGSRX/zNXwx/KeIE5viwPik09f4XP9ZbZa+a3VxoFqGmJOEs6JOSnFaLuwam9MPwYEaVmp6709VQuNm1/pZN4nr/9DFBKfPnGwOf+XTljYFJxYZ5rvksZRnlWvbuU0vZP2tvVeJos5ReJouCF440WdXlVEpJOSstkqXm7kLXrb+iE/H62bE8RXHrjNtKXk7Ipw26cpKrRHNVyqnMPjReyutSV2mFuhrArUSfr7rF2ROdWE0cqiac9B6zIVEb6vADdXmb5FdHxNRhY08a7+ZvmSV58+gW2jY56pDdiFHgVz75K+EvRX/usvMbqC0vgVKeKbYR1JCNo9JjQyMzrRmlBJDYTqFFQo/3rgMmo6wYLbLCKhvoUpBuaM9XnqJML41UKRuRTlGamDpLWXtgUsxMijtgp0Z+x2y8t3biLGW4NhbHSHfFM+/F0yFOXMZEpZwT9id87t29dv3ug7jQT2w6Ko9iKm2r3t0q/FCU+eoTYe1thuRzXCjPUWHUSErUMcNaaNQwO4s46VnKRuFJsZpSdlQnbJU3NiK05o5CztpfoPR1r01ZKuT6p2uSpZZVK9KVh77vkF2+dQUoYFN/sWadidVW1V3te9RlX48CZq/cFUxU2tS6skJ463YmNtmeeOyB6J0fvAA2cFsXWxYVNfVABnBnZJikemvIRtKgr9LuNBXI3yq2tG6hPYraqgqI9pBsSJyytPdZ7KkXO+9w52X8WUYprxSmbdV6TvGtRnmAK4cdBYsDnh7v80zC0F2+s8JqYO4Gg25oz1eeIjSv8AuuJXLJtB9098rGmKiUx/ttHh/tWcq5N74O043oJQYzKmV6k8M0BSeTRizFROXkG19X+22eQiz1Tyk5QoRaHE0yrbN1SBMrWmI3EWqpFq7PuCaIGcVLMEvZwlz31lBK160qbrM1rStABZc4BWSCXl2VOWRdb/qybz4rVtW/KyptYl2FnckuYnm1l0zDMXgBrGLHx0xdmsPcmgwNqeshW0LEVVp6Qi3F5tEtNPtcOgUVRM9KD1a0k3wHDP5cJ0w6xSjllcK173Jfcm8VmjhKCWBwl9paX5g0ROWFkzysHwzsFicq5bywPpnb40SljF6XPjApVDP5zZ4ZYTWyZ+6zlP5CN0h/JTTl82E+MEkpjRaOQbii/GEe+fdUpRQzk8IhTaQzk+vMOk4N5Tnj49T+eaRAKctKaaVrBNKF6h+p9ugSrSvARMqrsvuZX+H1ZT+rK1/PotJSDxygq2RDuua02u1u1KViplKafc7roTuQgNvuMHm91WiscEKL9dQHkfH7U6DYPLqFdv/HK6X5VCicniNxrvsmlJCe/aSdjAKlvFK4nlBq9KbZmU5+0TWm5Fq8qJMAXAH08B8oD/OFwrZj+pK2J4Z3u4t7ZeU9yvRac5XJP0rpIv6Wj47Nn6WcPYe5cJYyjr4luhG9xCSlzKP571LapyuTRyjtRdlb5SbPUtqf4amRy2djrnLAVJXnmGi4YtsPzQdXEMh37jyaPFRpHK9676iRusTcsgcy4/KS+5V2WE5+Ssw+lJSygN3bTZTSXqBSqj2lQrYGd6Hrslid3SrKV9dhzGon7WfrClCjOwRFvipXA0WlyS77YmiI48U8pRSrclhd6XdsoNolgxfAHLvp4vVWKeVmlM1qFo2RuiDMeSXY05piynRtpvyRAnagKDH8LKVFn/1KS66CUl4pXJPNG71trK4ZmXaftNpKeYCrSnnkrowlEwq7PujoO2B3ca+svEeZXjWMPSqltL/WU33SctIsZfWBSTEzmf4MbBamZHJrayFm/2BsqKky5irXGKrzYd7hzsjxlNJOaRZ/qscunDhLWb27VcxMprOXrSj/4mslRhRWnlOP4t2nSayjlMmDkc4wq9OGzn6FK1rHW6qUjRiugT6Or5S2X2SNv7hwMl0WO+A25aurwBSokHhdvZjBlDQHpTxwmO4QFGpVtrqqh1C67Pe7Glc+QymL8mYHIH8ZLFa72dXCRXKiUto1V/2tuFe6xhYQR9iMyUppT1yZwiFYtEjbTHvkcXU1EE/QMOK8mDoXGzLb7Q620mD02a+05Coo5dnR6DmSvum78n2jSZu+6zmVTgJwBZkwck8snNFd3E0PbfZBZXrTZinrMUkpl/42j/VJ8ynz2dbv95xCKVscTymbMU0pJ/42z1CULfFsZimNtj1642a8odSu82ZZCN185o0bUfPCBGNhH6ymlpmyw3tSyrIA1JdPoctii24TGby6Ts2GK8yxmhU2PXTZD3QVvmyskRSrff5FcizlljOn8stMUbIhpu6Vq72oc07zXho5O9pta8r+i8aQnLiqajoLGGZEg0QprzamlctvvBzOKn1rM80RpQSITBq5lw3zS5Sy+i+FjI8pSll9YDKqZv6TrSGMRh4O8S33wGS5JErpY5pSVv/lyTgzmapmO6pKOXoNaSjPMTFGh4qxWCmts/lNm1Udrt+5001CFnYpzl76T70aPluc1dz5LOUbf/fvHP7O330jOqSK0JoTbG6QNf7iwsl0PoZSjki9UMqRnEwpbacI9emssm8b4xpJt60J+y90MXqsJTaSka1rOiglNLCdAaUE6DEX5QqFMWZS4YwlSlml+NhkMSb+PM/0MJJp96j0GGR4S89YGqW0i0u0VTPUVBlzlTtnpVw5Js9SDvyQrPIcE1bnahRnC7uYaGgq7Byj1DC7G2JzyUs76yi90X5WbFq9PMdZSou9QKWMz7NbdKn2CkpZZcK9jpP8IdDY9FjV3J1SVlgrP6wqZY2JF22rZDWmnt/GXqmqsyXlkq5tB2w7GWiKXfOboJSnZCdK6c7QWk0TAqV232y++XVw7OXv3KBBwqVBmd5mobRtzxFqanOU6W0WStv2HMpziM1CeWMjQmsGAGiyWCnfaOm60Jf8Cy1HMBwy+LUJX8mk32r4k1WuZ++f6bc1rfJ7hQZ5VHy7GqyZvPmJ87LSl6kQUKa3WSht23OEmtocZXqbhdK2PYfyHGKzUN7YiNCaAQCaHGmWspGRF98ig18Z9zxkIXd385ClWX7vAHoO03vX5Ls+9gcNchU6Dx+omV7XO6XMPB+rXBFlepuF0rY9R6ipzVGmt1kobdtzKM8hNgvljY0IrRkAoMmJlFLNnpHBr86VnaUsQ4Ncjq8oR7tmokD6Wg2GGW6iPp9GtReU6W0WStv2HKGmNkeZ3mahtG3PoTyH2CyUNzYitGYAgCZbK6X/NVH+Sf1NKNzb2X428nyfpaRBLiT45MV1901Fq2Z8VV+IbzTUXLd/qRwe5qNMb7NQ2rbnCDW1Ocr0NgulbXsO5TnEZqG8sRGhNQMANNlWKd3Umc3d7UyFzCmrGT/AGtAgF2IqxDnh0Bxjd2e1q8DiLGWYPz+Du6kBAAAAwLJAKUNqOESffbry/ayXmx3qk3gyeFgMDXID2krp3nVVKpWy+1QCSgkAAABwJhxpljIj5poRf8+hyyzJ4GFbaJDzaCmlrLRUKQ29VV6/fTu5DxYAAAAALjdbKeUAZPCwK2iQFRpK2UtjSvbjrjxLCQAAAHBWrKKUfkYihX9SH04GDfJozFJKN/3bnYLGGgAAAADgErJYKcMDbOmcQytr9Ol+etsbWSasBQ3yqOiaKdWexS/vzkL2mGs2dQkAAAAAl5WlSunnH/IEsfvhx/Ay4pPLs/0n9Qu4qkgNx1dCw1h84u5I6lak5nEazRUmR/fQII/LPKU0xCZNrQIAAACcFcxSHhlXP+ldlOHfVKger/BGR19XvuoinTi5FZ5l7c2ABgkAAAAA549JYmUG2720yXCaCR+fVZ6l7BxJ0H4U7eo8upZNjgktrBiLrxxfIfIhtMYDabWpuasKDRIAAAAALh95UlrP8NdSSrWegPULfR9flVWUEmp4gexPavDJi+tODstKmd5y6efcXEnfwrpWla7ZG+bosw4AAAAAAHvCWYDO513OLxfq+xk9xh06M0Qpzw0hhI7uhA3fh6ks1LxMVbNungAAAAAAcMmoqaBZPirJX6KUFVDKXVCbPFxdKfWnAAAAAADgEuHy+cFZSo+3CYe4h3G2UvZOEWGWci8kTz9KjqWUhdYAAAAAAACXglHPUlppTCzAlenMkFnKM2OWUlbvaG09S4lSAgAAAACcB0YB6je7GsVIfkjSmIXVDWmGk5RS0NxuA5TymHh1nKaUQR0n/eJrZpgAAAAAAHAZ8HNIg3SZf10p581SClDKPeLbR35iykLYzTH6dyPpjKUgzoPXNgQAAAAAAOeEzfz7mxOtR8y88TWTizIjFAOlPCqVycO2UhqEVSb3T4sTr7+cMOTToQAAAAAAsH8agpdLnSgsHoubqJTrgVIeF3cXq3DF4+DviU3kEwAAAAAALjtv3B53K+JcpVxDQVHKI+O+QkhmFNfH/aIPd70CAAAAAFxSxMSjBqUEN1F5zNlnd9crU5QAAAAAAHvBpuhbqh1KCQAAAAAAcGWxaldjzGN0S5Syxtj1oJQAAAAAAAAwE5QSAAAAAAAAZoJSAgAAAAAAwExQSgAAAAAAAJgJSgkAAAAAAAAzQSkBAAAAAABgJiglAAAAAAAAzASlBAAAAAAAgJmglAAAAAAAADATlBIAAAAAAABmglICAAAAAADATFBKAAAAAAAAmAlKCQAAAAAAADNBKQEAAAAAAGAmKCUAAAAAAADMBKUEAAAAAACAmaCUAAAAAAAAMBOUEgAAAAAAAGaCUgIAAAAAAMBMUEoAAAAAAACYCUoJAAAAAAAAM0EpAQAAAAAAYCYoJQAABH781z8+Sbz3s/f2H3/1F3+x//if//7fnyT+8rvfPWGE5gtD/PjWrZOEOl8bx0/+5idrRahHAMhAKQEAIKBMb7NQ8rbPUPK2z1Cmt1moJH7jCM0XhlCmt1mo87VxKC1cEqEeASADpQQAgIAyvc1Cyds+Q8nbPkOZ3mahkviNIzRfGEKZ3mahztfGobRwSYR6BIAMlBIAAALK9DYLJW/7DCVv+wxlepuFSuI3jtB8YQhlepuFOl8bh9LCJRHqEQAyUEoAgOW8fvtwce9BeDHEpMJt3r13/XD7jfCizbsvXVy89G54UUGZ3jrx2pOHx559Uy1MQ8lbNd65e+363Qdq4dK4f+tw7e477733+q1Dc+VK3hrxzu986PD4C2rhOvHipw6/9IV31EIRyvRa8W8+/cgHP/09tdDGF54+/NyX/02y8Bv/9PD07yRLVKgkfuMIzXd7Hty7uH7Pdqo3bh/8HyMwPfFw6/XwYohJhQdRpteKX//Hh3/w2JtqYTl+/cnDP/6aXpiEOl9rxTd/8/DRf/n1v/zu888cHv9m9m4fSguXRKjH82NKA941fZc8Eet22CNxpJ1EKQEAxmDlrUAYvYqWaBYGUpeboZTJ1sXaMqWsZwYzlPJrz4QtFnnytaTwj//6zWcfC28lDjlVKY03hrX03Lrfv1Wwvgd3r5vxUS0cH7OU0qhdxs0Xw7u5UtoldT76O6/Iwjbi+j/0la+ny+coZcES96eUX3/+w+GYNR/+/A914SRC8z0qJlUNe+O5bTOyplLavK1HvFvL54rLSwvjhUUzlCYq07Nh1LHnoV9PlkulfOKxh0Ohnr//7BP+3W2U0nhj4JmvhIVzlfLlp8Kacp56WRdOItTjIirXf9vAXKM6LmIokS12jlJWvtDcwE51ZzR0VbqVUia9W/S7Wu+uYYadyPV7r8vVmquMrt563y8dtdxJuarSTlbO5hRQSgCA5eRZgr30dxdo+7fQuUpKUcVe69OP9y+Pq5QyvF5mGtmH88lnvuZf2sLd30tnKaXpLVVKY48SZ5LrzFK+8pVfiu63dJbS+uSnvhn/FlZ5NKU00ljGlZmulMYPH3r+P7q//+PnPyqd8CuPH37zef+3UYJeD7xSipcTIjTfzejT1rpS2kwxLhF5vKeUdBaT0dLCqReQiDK9N6///YOwwa89JKwyV8pk0vK3n/0HS5Sy2jxEk5CiaNuGs0fXfg7hsyvPUv7wuUcmK6U5s2WZMQ1AXP+lK1ZOn9WkiUoptqgu75Wtu3bYNSfbRPumdWKlnFGNElGl6cdHUa3G1l7JY5Qvi724gfmsPHFTPx4o1bldVVxoqiierNJWKmdzCiglAMBY5Bjs/7aX5kCSJehLthmQ4nBYSSlq5KNFXJINA/Xh3OzSPKUMc5VWEb/2pPuzIJY/ePbhxBtNyYef/YH7e5lS3r95uHbvQXi5glJ2E57JwsVK+fUvfLSXwLpSfvPxZEKyUszYaZzwNJEUm6+UmRNWZykLMV0pRZYvHNLED//lQ8ENpFeEl5dEKU0v8727TzRVv0v6uycmxKV8ziIuL7ZfR3ThiRcQQWp6RgsPT/66XGLksBPF4yplrXlUVDPTTt+EdqCU8bTGVmGp2oj7SI2aMlUQWzSNRwwEla0X9qQrVh846qStVDJ5VTOqUSIOpFWsQrUaK3tlNqHPlK0K/8Fa765htriOUupPiTrxiFNc2ko8hNmglAAAY5EX4nQkUJfv/Oosl2TX+jZiJAjEUTPbUF64w+68pz5iJaZnVNDx8ItvJsvjDa5Pfq1b+OaLD8dpSRdGRMMHlyily7KjBMZ7YqUZnl4plRzWlHJUsdRO9ZJZSvm9z/zcIx/8uYOaqCwpZTJXmc5hTlNK4QDpVKR4qWYvL49Sxr7f90TV7wp5remqoddXskZ7WbjoE+iOUuGJFxBBanpCIENYyYxMUEqPWlsMdb7qzSPKofjqofCthH85VynXu/FVnGjTKsSl2JyjYB3mDKbf5VVOX0FUBhBrjptzlLeet6XYkusDR52KhMxY1dRqtHWl6Ko0vjW2MqvVWNurwpmKVZFXcpt4ChxTPx4wda4+le+kqjS9lcrZnAJKCQ1M75rxbRPA2WIv96FHhOuvXRKQWUJMH3vEmGR61qSM0G5LjDq2Y3Yvs2GgqZRy6CqiTC8JPQ+ZRBTILoxkLlVKJ5CJK9ZnKSsogTySUr5wUzxIaSJ3xW8+HnaojCxsBFJLo1n/AqXs1fF3PpFYZaaU1if/6Rf6l0ZEe6vsVfORz9zpC8jo03eT5Rf56P/9/3w0/KnoZcAqZYWWMJgIzXcjRP/tk85cKXXSGbPVYtYYrg/9CjtKhadeQCKp6Z1glrLWPD7yf/5C+Evh1DHRSxP23tfAXKV85LkfqoWjItSjO19FfuWTvxL+UsSTWDl9hTZTww4BJS7+4EuZannc1pW9GOKgMFcpy4xe1cxqzLpJpPGWplqNzzwR/lL0tWd3W2xFviz27gZpDdy+FzMKy+CQ7Rl1lRCnuFS+fjbHNkuU8lyx1yZ5CfO4/pN0tkobCh/MywNcbeJFWbmcunybd3XSYEaOuUppSLqqGGbUbrg9LGE+ErOHOlHzjECGjzbpXHFAKQPdrbBZKHmz4UbaeMurj7JSjg+plObvnkVK2QmkFctIPv04MtZVSquRn/hGXGLvgA1aqJXyzpc/qJ63NIXDZ6fOUoqppzTdF9NT2hMWRGi+m5BkY36o9aRjpcwyVf+1JKOzfTf2zTS5H5UsjkaZ3rRnKcOu98xRSh+15lG7KbqglEtnKZcqpSe9qsfMW1xs87HAZVZlxubuDrFF02ZEI6lt3eytGgJiyThwzGxaS5hcjesopadajbW98tiO2ZNWftZh10BeahqoS0d8aVtdnyocaSdRyjNlrFL2FN9qlAe4YtQv6G7MM51FjsQ2R0xML1miCi8h31AVMTxXUaY3Pla98TXMOhbuZV1TKeXCBUppHE/dp6pnKVPVrNKtZMUbX7VPpjFxlnKKUlaeiDMh7mBMvWIMVW0IzXcD7HWglNqmHuiRSafsemk+Z7uw7pg2vw8pbCn5c0NzkaHxWpmeDaOOPY1ffG3FVKWsNo/qNxHZja/eMBcoZZVH7r2VlY8R6jEgPKdvCY62jayD2KK6tjcMTbUlUzJ8sNSA69RbYMLIA59ejbYbFrCF0zUMU63G6l61KXXYk2F3pkPmCUfaSZTyTKn0N0u5Y/gLhOr/buGkzglwRTGdJbFEfcm2XbLvX7pwAzkkZJiVTFDKMSjTS6J546ub1YyPVi74eR43eXjzvlhSj/hoZZMoiisrpVVH9Y989Mtbs5Ri1rEQx/h5nrFhNbJHCOdEpYwpvppfEi8HNGBShOZ7dOyYmHS3PteckpGvmc9NSXYNyvRWiqlKWWseQjVTh6yZ51ylnB+hHgOmPYSrujmnZRtJJ77WJK5ZbM5S33oyDBnESDRNKddln9VY26sBJvTuRqIuOfZ5WW83UMozxTcR3aztcFhsFrYP2B8GCIOlyawiJ7vKAOwU119SCkOI7W5d9mn/FmOSeSmHrtEURn0zCmqlLOxex+C4mJpe+H3XKtm0ZL9E/j37WcoH964lbjmkfCNiRaW04lf0SRMlpfTlS+SKKGc+7d8L/xGRO1/+YNiUJp2lbMR4pTRZfpERz1LKmCacofkeF9WLHb3RFTNyncEH6kmn7c4pQxeKfgfGoUyvD3cHbMLD139blQkhZzU9cm6zEv2ZqjWP9rOU/jnbIJ/2QcrQNk6olEmaJGg+BOhysGEKbSaltp6BZyntJ81ul/6epZSNwxk8hMCsaozkg91492tU4+CzlA3qvXsV6nVe2Kgp3J0IcS063h6ilGfKJKV0hW1Waq4pyYWgUh7gCmOHwKxTuIEtH0Tj1T8diszyIyplDbOTE5VycliT9EjbnKuU92+aqo6OZw2zIIQmOifUy/MwJVOsss5RSvtzO/V5yIpSlvyz8OSkC2uSnvRTc5WyIISlX3wtLxwRMoMXU0/JE3FJ6q+moVRYbejvgRyO0HyPiO3LhY7WG91ypXQDcUFZ24lsvwPjUKbnwv3Kq9bC4kL97KUP+wTmlFnKavMQTaKfhxRhGo8nfuS0s5TiippOcJnG0J1fc1FpXp9Nuyq0kBGILao2MLT1aHGyERYb8Dwqzb7G3Gp0379k/cge3YQDqVZjda9cl6zi/12xbK9aTKuuSvIg6kogWpfYSmUP56YlApTyTGk0etXZXMl+xLJNLTZuO4iudpUBOAdql107vI1Uu/nX7jjqu77ZMXK7YoysokzPRWmuMp2fHIh5Smlvar1166YZ+/wSK37Xyv9YyHilLMYcpWzHCkpZi8uglMIH0lx/5G/zeJ/8zQlWGZrv9pgB1HfJYkY+JVmsZqJyzY2RXVIftZXp2bC/u1MSwsLy/BdifdSWx5Anq9Y86t9EVGO6UraeohRU/ymRUI8O4TnmghxPtLjSpjZSIPngFMQHU52YsnVBsQEPkAxDKRMOamY1VjuXWcn44bVajbW9GqTakWtMuUpUjy7d+Q5TWOK3YnOV0gmaVG9lUMqrjWmF2Tegtj+EhuWa4+SrDMA5U/wS1PWaxaPCIIVRf4LKihG6ijI9/09QPvmaWlj4fddWzFFKO53o7NH+VM+1e/fdfx+I5TJQyhjK9FxsrJT1J+Liy5ozmOVm1OneMgpqXslf96lEaL4eO66N64w2mVuWRZk1+C5Z6JsTk0VbePos5USU6bnYeJay2jyiaqqfdKrHSWcpheekCX20kb55VDFjwfiBQyDWLOTHMmXrgmIDHmAFCVlQjU6NMo8yn5pwINVqrO7VIDtTym7N3VZs/fipVF1LK5xNlBIamBaGUgJkmMu3YtIQMvvaXRj1j62UJjafpbTzk1IRvVX2/5qItcr0HxexSyoUb5RVsZlShn3SbKKUZYpKGd7LaKqmSuKnRyqTMkaIZWi+29OnpIW+6dO4GsUk0mXJCUuTPIUyvT42e5Zy3TipUq6CSPpPS7EBD+BSxAojR6WFuC9zE1b8/mUes5SyRt7963VeVkpNXz+ZVTbO5tgmuhOldEcyuTVDiVbrFJRq2357kXLyznkiaJBwRVGmt1koedtnKHnbZyjT2yxUEr9xhOYLQyjT2yzU+do4lBYuiVCPAJCxWCm9wGg5zjPy/Ls3R/ggGfwRcaI4+E2n/34iLeanYi7XecmnjyLyixYa5Cx8f6/XjPxWIvk+QpyXbb68hHko09sslLztM5S87TOU6W0WKonfOELzhSGU6W0W6nxtHEoLl0SoRwDI2Ewpe4pvkcEfkVFKWT6P3rtWvvHmFNAgV6Hz8ErNVG9ByTwfq9wtyvQ2CyVv+wwlb/sMZXqbhUriN47QfGEIZXqbhTpfG4fSwiUR6hEAMlZSyiKNDF7fmEsGf0Su1ixlmUYDo0GOxFeUo1wzQTiDLsbGE5Yneknd7hVlepuFkrd9hpK3fYYyvc1CJfEbR2i+MIQyvc1Cna+NQ2nhkgj1CAAZW89SunkM/kn9TRmnlJbkdDjO5VlKGuRCgk9eXHffIFVqxteer8kwY2mvDGqu27/cxw8SQIYyvc1Cyds+Q8nbPkOZ3mahkviNIzRfGEKZ3mahztfGobRwSYR6BICMbZXSFbYZp52pkDllNeMHWAMa5EJMhTgnHJhjdPUW8VWazlKG+xpGfccBAAAAAPtnwxtfXcl+1stNYvRJPBk8LKbRFCV9M6NBzqCtlPoUdN7oP5WAUgIAAACcCYuVciQup1R3Ufp7Dl1mSQYP20KDbBJuW/XIexBaSll7llK8tEtu307ugwUAAACAy81WSjkASgm7AqWcoZSu0tRMb3I7sYdnKQEAAADOigVKOfU+Q8H5/gwM7AEvMyn6cd8EGuRYhpUynaV06hgE1Z+C1hoAAAAA4PKx5iyly8sH72fzeWdajCwT1iJ805FOgg2LEA1yHLpmktpL5jYdwcyzr5+CdgIAAADA5WdzpfTJpZ4yUv/MwFnhqqU3nPC8maVhLD5xdyTJt0jN4zSaK0yO7vFWk9dGtXFevQa5iKZSGqRVJjO9sUlTqwAAAABnBbOUR8YZS59buyoSFO/G1FM6fV35qot04uT8hyk1T6g9ZikBAAAAAOZjzGXks2DbK6VFm5WQrjMjqZPEdkom4/CV4ytEPoTWeCCtNjV3VRFTwR3tBnZ1GiQAAAAAQEBPZXmCoZgM+QRKCRneG7sZM++B3cyk15jMA9NbLv1pduqYlm+tGQAAAAAAYDRGLvRcF0q5D4QQGvxcYn9i1MuO1BXjS/V0X9U8AQAAAAAAJmBtAqXcJ+nk4RGVUn8KAAAAAABgFEZMLq5fqHseUcpd4KVxQ6XUXy0AAAAAAAC0eOO2u9vR+oV8KA+l3AVKKY/4LCVKCQAAAAAAU7GG0jtFYpUo5T5Qv5rj/TCcs6oEenX0509KaSKofs3x4UllmAAAAAAAAC2sX+hfY7FW6U0EpdwH+ldz/AykINhmqpdeFyPpjKUgznDqDQEAAAAAAGwBSnlU8slDYZV6jlHMWAqrTO6MFVaZfGegpkMBAAAAAABG4O+RLMIs5S5wZ+jojzj6e2KzxzIBAAAAAADmwI2vu8HNK448GXNxM5/c9QoAAAAAACuBUu4IN1F5zB/OcXe9MkUJAAAAAABrgVICAAAAAADATFBKAAAAAAAAODooJQAAAAAAAMwEpQQAAAAAAICZoJQAAAAAAAAwE5QSAAAAAAAAZoJSAgAAAAAAwExQSgAAAAAAAJgJSgkAAAAAAAAzQSkBAAAAAABgJiglAAAAAAAAzASlBAAAAAAAgJmglAAAAAAAADATlBIAAAAAAABmglICAAAAAADATFBKAAAAAAAAmAlKCQAAAAAAADNBKQEAAAAAAGAmKCUAAAAAAADMBKUEAAAAAACAmaCUAAAAAAAAMBOUEgAAAAAAAGaCUgIAAAAAAMBMUEoAgEvAd//yu+cXD//o4TOL5/7f50bGez97b//xF3/1FyeM0PShzt/85Cf7j7Cva6Da52YRNg9Xj6wt7Df062NGqJ0UlBIA4BKgZOw8QvnYGYTyxkaoEXqfoRxv4whNH+ooedtnhH1dA9U+N4uwebh6ZG1hv6FfHzNC7aSglAAAlwAlY+cRysfOIJQ3NkKN0PsM5XgbR2j6UEfJ2z4j7OsaqPa5WYTNw9Ujawv7Df36mBFqJwWlBAC4BCgZO49QPnYGobyxEWqE3mcox9s4QtOHOkre9hlhX9dAtc/NImwerh5ZW9hv6NfHjFA7KSglAMBxeHDv4nD79fBiKUrGyvGtxw8f+/zX//K7zz99eOgrX9fvluP5xw+PP68X1uLrn//Y4fFvqYVZjN4N5WP1+Pn3H973gT9XC9eJD//h+w6f/Xm1sBx/+v7Dr37gw2phGsobG6FG6OVx/+bh2r0H/o9br+t3y/HO3WvX7z5QC0Uox5sf3/rU4WNfeCVZ+MKnDp96IVmiIzT9Cq/fOly89G54MQnTMa/fm/XJrXn3pYv2MSp5Wy9e/tzhkRffVAtnRtjXQd64fRg6L6p9tsK0j2bblvHg3jXfd2oRNn8czFk2PTa8kBRGkNdvHy7uPQgvNuTde9cPt99YeVBziDUv7JX9GkY0pPFkbWHFuP+5w7UX31YLq/GjF64dnrmvFsrQr+sxYYwQI4uMUDspKCUAwHyq2YChNPra8mUGEoVUxqzaKay8VVzu6195KBSKPPT57/l3i0op1y/f1Uqp1/z083b5TKX88Ad+9fD+PxVLosINKeWff+B9Tdn7+c8e3veHH1YLfWilNKsKBxMJe3VkpTR5bdheRjqiP7h7PSwP3Lxvlg8p5f1bh2t330kXnq9Smnc1fZa5PHmdhtGAEVl4KQ+eoZTffvJw7ctvqYXl+MZToWYOT307/u1N8mhKaa+KGmsUhnlKKc+06whxedK2TfvX9N1qG6VMLv7iSHellP1OmobXV+3tNxpKaXapQn5Q8ez3RzFLKXUrcrvUr6HQkMr1Vq15QdYWJsbbd/MGb6rmNftuWSmtOgZ8MbF8tFKWR5OugxTHCLOw45bcDEoJALAFrTGpMPra8qUc0Qyrk5Syi+99/iEnb+FlXSlTtTMa2VBKs0R83KwzFmjNUsatbK6UVgubBSYq5ft/vn8p40SzlMURvQ+bOmynlK984WMf+sL33d/f/8KHpCUaabzxgv/7hRuHT33L/GGkscyH7r4yRylNptiiqW0yy5yUvDpMbh20R/mhWW3X/UWZFHsdGKEBJaGqXC4iSt5MTFBKH28+d80oZXhpTNL/fVSlrJypUg0oVPt0PhnTX9sX+vaslVKH7FaTlVI0IXWOak3FjhTx6KxHJcRBJHurx5aZrpTVXRWDTlqm1KSF+OnTN97WTMmwQvtuWI9Y89DZL9N/sP+j0JDG76RGtwWjiI/d/ZH72zjexQt9E3vw4mOdH4oyhbCSWVVK641i/dIqJyllH/0AISMfI8ySvr84HY3dSvaUPkLtpKCUAADzaY1JhdHXlt+7UnYr6ZeItbWU0hQLby1QSs0IpbROaIpZFayWuQxKWZhF6VlBKY09rqCUwgOFQ5p45e6HnCiav4V2tmLhLKVNTwfTwcgipSwn3wbRnatd2GXPw/foWjkKWa9NsnvaH1TyZmKyUn7jqcOTL4e/jV5ee+6/278vhVLa6fq0qYslA0ppSsbuMFkpza4WvMtQayqZ0th6CEvGiE1HWY1aVHfVrKo7EaKM3JmVlVJsJa5Qrnlar+zoV9uvwSzRq3KXiyJqJzN0W/j2rd7rXnvm8Llv928ZP+z0T5QpRBTOXCn1ErOJ3lrnKaV1Ra+UdgiIJB1Hjw62H/U9AqUEANgCNXC6vFBy+ZTS3svq718tLGkopVjnUWYpA7qAIQqhK1ayvss+S1lQxC5GKqVNmq9nX1dPVUoxM9lNRYaIL9XsZTpX2WmnXz5bKW0aGhjKCAMyy5ycvMbkW/XfmHPX1hmWD3Vws3uHi9u35ESWpXK5iCh5MzFRKRN1FJ+9FEpZmHiP6a85N422bXPoZHpzklKK8yLEzFJpKoWjTkpmzVi0cEN813xqmlJWd1W0WKmacmfWVcqkQtzK3Uux5qGzX6Rbj1hDWSnHeW+GagtiZlI4pAkxM5mqpo8HLz7WLWwopXFRNb0plsxRyqCRhW6SjBFZZ+mHFROxT4kItZOCUkID0wmTO/4BQNEakwqjrytfZpZSGnmTQriGUvYr6ZeItdWV0n7qofhg5Vyl1ESlTGcgvUwWBS88DJlo4SVXSpUQTH+WslNH/dZopTTGWORDv/fkh8KfCq+L1ieFeb7yhY/1VtmrZnVKMzT9hJBqd7lpl3lnfdDkSJp+LBudvFZ76xPPVN5I+3tyBajKgNvV8Jb826Cy8BwlbyamKOVbL14ThfUdsJs9S9lVkfVqT/VimLbP+bOUKkserZQuKSpw8QdfqrSI0DLzs5+uKmnA9i1lX/X11Kju6jNPhL8UfotyRDPbHaeUFdJeKdZmkZ1rgVKKCunXUFbKCulO5nRNwOhfkWs3PhP+Uoh7Yk0IpYyW2K+zE8t8elOY6nSl7CYbXQOSn1QDgRTIEPYjEZTyahMvzTnyctCNx4p48UIpAVrIAVhTGH3no2TMh3E2Q3Q8a5iB+Uppl4iPJ9ZaU8p0ecVs81A+FqIgdQPPUg7GakrZkUyWilDe2Ag5PJsBfgxdEpBn0jbkSpJ3kwmZ9LOjldKHmJlM5xjFzKS4AzafsZS3y06dpQxJocxKe0L+I3qiWVKVsYnJq/A6M2KKhF6sJ3c/lzGr7m8HXFXM7rnaGTt8h63kq1UoeTNhlLJKf4OrjdQnrUMePveN/t1LMUvpE9/pz1ImPcLGxFlK0xS7QzD7LFpdtan49hCPLnnXviUHkbyK4pKoRsWOkFHdVVM93RpEmXRnRBmZK6b7NhqxNktXV3bNdvnEXulJqs7WUkdjVRM3lDYEMTOZ6p+YmUxnL0MIpYxRmqVcSyltI4+/x2O7hpirHKWUzFJCE3cxKvel4luN8gBg0dmAJM8MFqBkzIa/69X4W3+fasXlJiqlCauIHfLdslLaO2PlxOYMpQxTi0WMTKZK2SosEDOKl3qWMouqUvqBP0kXTGaQZs/xq2vz9zSlnPTbPD7as5RLnqUcwBx3VcYm5pQiFU6Sb5mjJ+mydUK59SxlH+sDwyh5mxtKL01cEqU0YVt4h0yLzfJy27YPLavuM00pRfvpvChQbSoOf+odmW0mg4hNvdRquwLmreSzA1R3VTht2h3kzojDEeI3d1Cr1NUCpbQ7I2qjX4NsSLbMCOqbThpCKnjDv81j/g4byLBlSkrZfzZbMkUpxUW+D9eM/OeVUuadRfYIlBKK2OtUpee4t/SVolEeACxZNiBIR197PR9BN4RrlIyZMMLm7M5o3sBDjNOVckqYjao1zFDKI8elUEr3RXKZNAOeopTLInW8KIHJVGTyMhdFq5E94lPTlHJq9zHlZf6aMC15Fcl3Os8jUmQzVjby7Pa7jnriWz0Kh5K3fUbYV096VUyYp5S1yLJkF9Yn8/x4mlLGZiDahqXaVKbjU7KOuCqzXG5xiOquxjapTG+CUtYbbUJ/TpM66fdHrHlatmmrKBku+zWMaEjjSRpCnEUUDmkjzkymqtmOTCmzJct/nmdU2ClN/fM8/SCCUkIRd5Eq9TR7ETlcXPiO7a4jkfV6JsD54fpOAT36lhAD9jBKxuzEYD85+b3PP+Slrq6UYbciI5RS/fZPsaT1yX5VYuECpTT6J28rNcpXscGff3/R/Yz4SUV0YdZZxGyopJQF7C6dZpYyF8jpSplc01PGzVLOfZayEUtnKdvdx7y7XClrvXvss5SBmL5PRaX7OUreZkT9RtnNnqW02FM5wgRU+7SR3cUawrQA1bbdrYDFb1tGK6VLogoMPksZyftitRkXRhCzAyOVsrqrw89Seq7fu9fv6q17o2YpB4Y8u0v+YO1Wwrmep5R2Vbpr9GuoNKQJNS/omoCRvSIjn6UsRq6UzhvltKS4h3a6UtqRoobqCGbU6PuL/XJT9B2UEoq4S0ze0+xVwHUt0w+Ty0GlPACMYWB8XaCUvUN2EQxztMuJqCul1UXxlt1oao+qQB+LlNL+SI90SGuDmSK6mKCUjdBK2Yg9zVIqzJDfUspaTLnxtf7PhAg5TP9lER9qSnN8hKZfod198vTR4dLxScmrmFpJt2jGx653izIlRMkabhQusoFSln7O52g3vjY4qlLaBlEq5mLKLKWY7lMNqdpUekoi5BOtYvspjCCm8EilNFR3VXxVIcq0EOKndkkuKexwii3g6YvNUEpbY4Xq7ddQaEh2K1nNFxdqRCuo3N2azEyq2UsfxQcpTRSU0oQzSU/yTOZ0paxGcYywVulJBwWU8goTe2yTvr+58n2ncl9Q9V3dXenGdnIASLGdqzW+VtKOMkrGyrGmUtr7aR9/Os6FmnU+/rR4brMRS5TSCuH739///s2ff+B9v/q+yj84aZSywqVVynZe245jK+Xk3+bp4iRK2WJC8lpPvsVKRBmDGzeHSa8MQ9eKGtLcWj/MIxC/wRM+dZ5KOSWmKKU5v92ZSr9KGPa02gHKBmnKVDHrnKSU1V0VfUeUaSHEr9F05zRjsebRvbJMv4a8nms7VjsjAtEKjAGu+Ns8k+PYSlkLlBLG4S5e6ksaZ5X+muWGxoWdHODKMjS+7lgp7c/w+DWYVZl1ft7818nkqDXPVkrjbGHi0eri+//wA0Em7fLcKnc+S/m7v/HRw//x3O+KJUmoEdrEvpVSzEyqqcj4Us1ehrjUShm3oj4Vc/SKP0xi6FpRQ8nbjFhBKe2/PtIqHPZ1kBH5vWqfNjZWStEMVAusNpWIzalUxuUXSt9rYgqPbmzVXRUtNjbjNuOUcg6bKKXbSlbzxYWa2ArEzGT6L0+Kmcn8J1ttoJQAKCXAEobG2kk5sZKxcqyjlGZJ8nFrlWJyUr0sxCyltGonFdE+06h+5VVZ5RnOUob9z1G/8F6KIyvl/DBKGY4io62aoelXmNR9EpYnr6tjrxUVmruq5G1GbP0sZYP5SllD/wvvtZgyS7kU024VU5rxFKVck7ZS1hijmtsopWVezWdtYUZYpawxXjVRypGgLsfA1aqi+XVU3t8Gv785U2iQsDuUjJ1HKB87g1De2Ag1Qu8zlONtHKHpQx0lb/uMsK9roNrnZhE2D1ePrC3sN/TrY0aonZTFSvlGQVs65PcT9nuIAsFwyODXJnxvlH5F5E9WuZ69f6bffrXK7xUa5FHx7apVM/GLDPl9hLubWkDdTkfJ2HmE8rEzCOWNjVAj9D5DOd7GEZo+1FHyts8I+7oGqn1uFmHzcPXI2sJ+Q78+ZoTaSTnSLGUjIy++RQa/Mj6Dz+f03Txk6a4Jrwp6DtN710nuslgXGuQqdB5eq5nwRUakb4F6Apy6nY6SsfMI5WNnEMobG6FG6H2GcryNIzR9qKPkbZ8R9nUNVPvcLMLm4eqRtYX9hn59zAi1k3IipdQ3WJPBr82VnaUsQ4Ncjq8oR6VmvDf6yckwLRm+pDib7yZOiZKx8wjlY2cQyhsboUbofYZyvI0jNH2oo+RtnxH2dQ1U+9wswubh6pG1hf2Gfn3MCLWTsrVS+l8T5Z/U34TCvZ3tZyPP91lKGuRCgk9eXHeiWK4Z396UnDvU7CUVOwslY+cRysfOIJQ3NkKN0PsM5XgbR2j6UEfJ2z4j7OsaqPa5WYTNw9Ujawv7Df36mBFqJ2VbpXSZpc3d7fSXTD2rGT/AGtAgF2IqxM0xDk50X793r9Py+H2E/5Sk+TNRAAAAAHCJWKCU2XNTZfrs05Xvs0w3O9Qn8WTwsBga5AYMKmWKr97kJthwmrgJFgAAAOBMONIsZYbLRNVdlP6eQ5dZksHDttAg5zFCKcNP8oSZyfw+2HA/digGAAAAAJecrZRyADJ42BU0yAotpVTPUnrDNC+VQ6KUAAAAZ4h+jvYsQj1GePIIdb0/VlHKMDuR0HxW6nx/Bgb2AA3yaLSUMlRjMkvpSoYbX/2nwo2v+ewlAAAAXGKUjJ1HKKM7eYS63h+LlbKYIA7fHZc+SdXMUwEmQIM8Krpm0toLlR/pzFxLPsYOAABwZigZO49QRnfyCHW9P5YqpZ9/yO9hc/MVaZru8UmnnjLy98KVyl9+XFWkhuMroWEsPnF3JHUrUvaYlLvC3EbooUEel7ZSGopN1FBbDgAAAGeBkrHzCGV0J49Q1/uDWcoj4+onzaHDs2TV4xXJt6OvKz3V04mTW+FZ1t4MaJAAAAAAm6NkrB5vPffY4anX1MJ6/PC5Rx577i21MIuXbx4euffW5JUPhTK6k0eo6/2xyrOUnSMJ2hMRV+fRtWxyTGhhxVh85fgKCQ+huTk0+bdyntrU3FWFBgkAAACwKUrGbLz2VMirDlLzmtZnPnLz5WRJppSdPcYlYmFt5Xa5JF9DMZTRnTxCXe+PVZQSaniB7GfMgk9eXHdyWFbK9JZLP+fmSnrt6bwxXbM3zOYv0AAAAAAAHAklY84nH3nuh+5vo4X93y2l9OLXl3SxXCnt1tVH3IZGTH72Lvfg3jWbbNt027y8n01GGG7dF+53/6ZddO3eg36JjXwSI/lUt9rrdx/EhUmEujb0dzXa/L8wmxJtwlO+BU/MdXXIOapuf9NVlUApj4kQQoc5be6UDN+HqSzUvExVs26eAAAAAAAbo2TMOJ5Uu7fuPdJNP9aU8mXroEb8nAHGAkuV0i7Jy9eXJ9G7nFXKaHrW/ZxbxrAOmRa4dt1YaOKZTimTJc5Ur919J10yVikTL1B32Ll7GOODYHbL1y/souSuRisayRLnFMmqkg1VQSmPSW3ycHWl1J8CAAAAANiOVMaMH2aTjYenXrZ/l63PWKR7N4RRxDBd2VJK/8FARSmzPelCWG41epcbVMr33rl7rZdDr452SVosU0oTqYuuppSpLvq/nT4kYiLLdFhPEQ6JUp6c5OlHybGUcvh8AwAAAACsTiJjUSD7MGpXU8pmXJJZSqGUD+6aLP3mffFHV6aklImLHkcp+xlLNXVZVkq1NpTy5MxSytQV7VkMJW0jjGe9aJ4oJQAAAACcgETGhpQy4CYJ7WxhladeXqqU4VeC0o/4fVB7WIje5YaU0tmjLyAnJ5VDFpUyXdtaSunEwZuCk4vgI8ohi0rpPtv7C0p5eubc+BrU0TcLKaWJoOo1KMMEAAAAANiOVMYm3fgqIhPIxbOULsRvz3YM+6SJ3uVypdR07+Yl44/0HFMpMzovSJ3Q7kJqECjl7rHnoKSOZSHszpZ/N9It92sTxBZQ2xAAAAAAwPFRMqbEb8TP87jIlTKLmUopVxv9diB6lxuapezDTlfKH3pNHpU8plJmN74G7KSUdAQrGv0HUcrLQWXysK2UBmGVyWkWVpm0G18+nw4FAAAAADg+SsactnUTlfLvmvV5zcuVMlti7FFh1rYXpbQ2lpP+bI8sb8LeKBsXrq2U3jI0XeGiUtq1pRNXKOWpce1q+DQsxN8Tm3/HAAAAAACwAUrGbFhz88ibYJtKqRaayCWzFDtRSvXzrS5s4fAjPSWltB8RP+GzslImc5IBpyd+xquklPYjYj4MpdwFbl6xNhO9Eu7+ae56BQAAAIAToWSsHg2lrDBXKfP5zCKFnemid7lxSllebqXRm2SmlHa1x/l3KT1ug9ltjO72RmeSmVLm5oJS7gTxTcBxiM0CAAAAAOAEKBmrxylmKedG73JjlNL5YXZfq4n+N2CdFaTo8msqpfPDkiO46SirmoXbYnV5lBIAAAAAADZAyVg9tpulXB69y7VNb8WYpZTHBKUEAAAAAIANUDJ2HtG7HErZBqUEAAAAAIBFKBk7j+hdzpqeo/irPCuF+yEfw1ildBzzX3zobtRFKQEAAAAA4MgoGTuPUEZ38gh1vT9QSgAAAAAAWISSsfMIZXQnj1DX+wOlBAAAAACARSgZO49QRnfyCHW9P1BKAAAAAAAAmAlKCQAAAAAAADNBKQEAAAAAAGAmKCUAAAAAAADMBKUEAAAAAACAmaCUAAAAAAAAMBOUEgAAAAAAAGaCUgIAAAAAAMBMUEoAAAAAAACYCUoJAAAAAAAAM0EpAQAAAAAAYCYoJQAAAAAAAMwEpQQAAAAAAICZoJQAAAAAAAAwE5QSAAAAAAAAZoJSAgAAAAAAwExQSgAAAAAAAJgJSgkAAJGfvffeSeK9//bfziz+xxe/eGbxP//0T08SoWleBn7yNz9ZK1QH2XP877feOkn8r9deWzF+9tOfnipUg99zhIYOkIFSAgBARCWLm4XysTMI5WNnECq53CxC07wMKC1cEqqD7DmU6W0WygkXhtK8LUM1+D1HaOgAGSglAABEVLK4WSgfO4NQPnYGoZLLzSI0zcuA0sIloTrInkOZ3mahnHBhKM3bMlSD33OEhg6QgVICAEBEJYubhfKxMwjlY2cQKrncLELTvAwoLVwSqoPsOZTpbRbKCReG0rwtQzX4PUdo6AAZKCUAwBTeuH24fu/d8GIPvHvv+uH2G+FFm3dfurh4aWDfVbLYiNeeOXzu23phFvc/d7j24tvv/ezbtw6P3f2RfjeG8rFG3H/icOtP9MKp8eBLv3x44j8kC//ktw8Pf+mBXLIslI814r98/B8e/skn1cJqfPIjh3/08f+iFtZiUuGhUMmlim984vDIZ++qhePii08fPvjlO2phjNA0j8OYTtHx+u3Dxb0H4UURpYXNePmpwyPP/VAtjKE6yCljqPMq02vF/d/96C//7jtqoYtv/fPDzX+nFv7bm4dfuntfLYyhnLARP3jmFw+/9qxaqEJp3pahGvyeIzT0UQz3l5UobOj1W2OHxRFMOZAH9y42zRC6fRufmUzKYaYURikBAIpYVQvIS+oplbJoj3qhSZHDbvfcer1/a6pS/uiFa4dn7quFPrRSvn033fCt1+zyOUppfa/Ab99375aU8j/cCmUylDd2MVYpzcIaQ/6pfMyEVcfAP/yjf5Euz5XS2GAgKVy2xFj48KsffzpZnhX+T/8kLTM6VHKpoq2U3/vsB8P+9XyiX+GRldL02ZyuF+tOYTLCUEJy23WheUpp1LFHOuRGSmm7cMrFCw9UmRCmkwZcn5XLpyllXQWPqZR/9mth7yWPPvMn5q3tlPKtO7r9XNz5kSqThWrw8+POE4+ErXpq3cr0uBr/7Bu6cBKhofdkBiVc7tIppVlPRHxcr9+sXNNXwvGUUl+a0ouSykxK17FwRHkOU7hCdsc7JeFBKQEAcpxPdiZmx4/u70lX2KnIgao0HI5SSoVZZ58xH1cpZRi9DDnoslnKkuZNmKU0H0+8sWqedoWTZilHFFY+5nzyI//Jv7QGGEWxoJS2QLlwwRJl4S8+/Uf/SBhjXnhFpfxX/yxUX5lWbmoM85HPfvHLvxCK1nNfG6FpRkyb7zIelcCZ7im6re4Xpc473ClscjZXKX/4nEnxn3qt+HKyUooeZ7qV/8rGhelcXSdVvdK8zPrdgxcfyxd6n+zWaf8WVnk8pTQvQwNICWWWzlIayZytlK89dfjcn/m/X/3c4cZr/VtGGjtL/NHzj148/3ZYbuLPbhiHffGt7qULUyZfqEI1eBPf+MTh6X/l/zb6J7qS6XTddzGijA33rU3aj5xhTrl3IN1WKUJD78kMSvS74f6iSD/rO52j1K/NH2Xc/hS6fxvnYPEjZosJzQORF5asQo5C3EpDKUUFSkqXwRRx4oYLR1BKAIAMPSrMvMJOwg6Q/ZrV8BaYoZTJoL5YKV0yKqgqZcxxt1HKybOUhTiuUn7yycPhyU/GJcbreo3MlNJqoSycFNCWqAv/j3/x8V+VOrrRLKWb94gTj+24++VfkLnv1FlKkW6a/th/3ZO0cKGdHebd7qv9yFGV8uWbh0fuvZUsNKby2HNv2b+nKqXpfZ3mxa9sbJhO2k08ijI2ut7XlQxhbygQRmrDdm3ZnZOOv1Ap/92nfV8MbHjj6wKlfPvFi84DhUPaMN741Kv+b6Gd4WVimCHMqpJieagGn3QQo4W/8MT3urfcdzFeEdNOZO2x1Ins8gFLFLGOUgoG+ktK9auidr8uMlUpTXl1HRAbHej4yTUkq5CjEK973b6pzCRetTIGcxj52SkJD0oJAKCxqafIUw1xvJlyhZ2CGRiSIdAOzOk+uNFUD3t+YW3sVCsZzp5XmqU0b3U57iKltLenjlXKcFvsiJD++ct3v9MtP6pSSs3zIWRPK2VeuGWJRlbTO2Odvka2UEqT3YatjbFKO52SFJuolCJpU4ljfFlI7GxPMbguEP72DHSKmGCdVinjzKRwSBuxJ6aq6d/K+l1pljKXTLnkeLOUIoRSmo/3nEop48ykmooUL9PZSxOrzVJGtRMOaSPOTKaqqV/GKHYus7DGdKUMH4x0XXK4v6SY8sFk1FDV6Nd2jOsRb6krwyCF8rm2lUmH4MKVZ33M3nb10+3bekqZ5D8oJayEaalJFwW4IojrdSCOcMdSSo0dKbVSui5Z8szy2Gl2NR1U7FDh0SuJqGRxllJajbyI85nzldLYY2J93cIFSml98tqX/nN4+Z0vXevXf1SlzGcLrSUKlFJqDzSWWFPKObOUFZSa6lDJpYsgk2LmxK2pIZZx2kTmtaOU0naKEhdf+oM8r3XE9t91KNuJZH+pfM8iulXMEduZpUVp4So3vto+WOQzT1eOOpHD/OOFZykzF5UGu4lSvn73lw8f/devpwtnzVL+0W8cHrrxA/e3Ucpn/sj+MV4pnQGWeOpLnwt/KVJdXPYspbt5tcS1j5uWU6IzwGmzlFFZp0Zo6D0lx+v613B/8cSBSfHEM+1+rUZJu55uZ/prRal3FzDlq4N++0Bylzt6hmD35yJ+KbauUppLnzjYKQkPSnmm2MaUZ412KE0bR/JNbSR8MC8PcCVojS7W0zyjBsu52N6X78Ph1u3kCRNLRSntfuo9FGNkFZUsVtJZm2vWlDK10CWzlEb/fvnaw8IAXUy78VWpplHB9FbY+Ds95q2AllgXdmeSKc2hW2oTHysqZbekMEs5QSndkjirOfws5exIk8sghPJprj6+8Qn3Xi6W9vHL4pzJhFnKNGcV3UEkc2lTT8cyNz7271Y6hehWMaly63HU+pHSQhdGHXukQ06bpTTdqnx3q7BBUSaGWlgscwSlLGCNsaaU/qFK/dZCpfyT5x8SShn4jT+ThUVIxxNTkeIOWBtiZlLcARtD3SWrXlZCNfjW3a3dVKSavfRLRj9LeVylFEwYJUVPTK2m2q/TYpZ4QRBXiXG4y0L8SBzoPbUDycbfrEJWx1SCSO/NIbt9U+5nD6eulB15FaUrHyisQCnPFNc3RihlT/GtRnmAc0ZfVd34FEYydeE+CnaUKvVHO3KkY6ohG9L8x0vDSfbZAipZbERZKe39cvLxrflKGWTPTiQmWlhSytHRVsrWxGOilIUfjM0i8TE5c+jDyF6nkctufO0WdiQ3tR5RKSeH88xiIjtJKUUeqbI30ze7bisTSpvjFkbDsJJKpxDdKnb5+KkaSgubMU0pRXcTsmfC9KzuG5y0zBi69cjbXPMlk5WyHmWl9FOU/9b81937uuzG15pSTrnxdepv85jyY6g9Uaka/Izf5gnhHFJQ61bh+6AiJQWNERp6j882U7quN9xfJKLPmg+K8avSrw2qa9uBu7sgqJLjMNvtkQNo9UAK15ZjK6WtcLkz3b6pzMQWqytlbQ/NW+pTjcIZKOWZUurkgXLj8B1JtT+38Kh9A2Cf6MuxGFGmXGFnURZCM3R1ia9IqS0i9zW4vl/zxnWVshQ2kU09c65SCpNU/pYopS02guiKs2981Uqp5k7zSH2s8PM8vfst+3me8ZE+Y1kltVkRMrMM85BDhKzX/TZsO1VtRGiaAdMZQ+9Q7Vm8jGUGGVbKiLgOVFBa2IxJSilmJoVDmhCzjqlqTot1f55nWrzzr38pzE/auUrpkEtvfO1jolKKmUk1FRlfqtnLRZE2eDEzKRzShJiZnD/NmMW0VYWGPorh/iIQ45oZZFNRbPRr866/1FjE0DxLKWuUD8T6ZJ4MmCH4iBmC2RN1XN2+qcxkhlKa5flhTkl4UMozxStl4XvZsiK6PmnvzPYt1faTniP2DYD9Ir99lH9PusJOp+yTtofKjdoO3l/6i7lvmUr2nKCSRRt2vqJMZo/5Y1qzlNKJopiK/M93xe2vA7OU5rMDTznKu2TTe1lHK+WYUD5mtVDONApzy5QyLWD/bv4jIlZB+wLGG7u/58tnOVRyqaI8bSJDT6FIRs1SJmOTYMyzlLUcq9IpdqSU1u6KDD5LWe+5sUwfpofKaUl5r8EspfzWPw9b0qSzlK7Yp7/VL3E/Dyt+p2drpVzwLKX9ZdcaI3/xdf6zlPNjBaU0HbPUiYb7iyfRQsnQs5QNzC7NUMrRB+JGap1jO46olDaHz3av27eCUhawdVLKYdwpKNUqSgnTlNIVDu0saVKV8gBXg5i/yq50RKV0o1S+8uIWY35cy33HZ88JKlm0YRLTUkKZ3viaz0/6mKGUViAzaYwLFytlJY6slCacVXqSmcCCUpqwJulJfzVnqVJmP+czOlRyqWJYKathktqxN76KNmxavszzRJs3XSbP9mKXqWMHwRpmW132Vkc64cvFBwoDRianzFKa7lO4u9WEmJkUZQYiv83VhzVJj/BJv3yWUmb/KEh246sVSOGTeuF4pbT3uFb5tWenzVJW/5kQMTM57iFJF1Y1J/wjItW7W4X7pbOXy+L0SmkRfTa1wRH9er0RedyBmJd1X111fwS1jXb7VswTiuiSdT02jF8tSnm2OEssoxqHK9l3ofSLCtuCxzcmgCvBlCvsJKzBzlnzBKUcg0oWbYxSylrMmqVsxkZK6WZKh6lvS/lYI8pKWYuyUobdcQSldL/smupoV/jyKqVIN9NGLjK58lcndrCb2Sk6uuytjtLCZkxQSnF367Tf5ilHTSlrcTylHIhZs5SlmKSU83+bpxzTlHL6b/OY7jMG742TChciNPSUiolNoPpV0Zh+vd6IvPxA1t2fESxXyiYoJYzFtJVsGt1Zpe/PKCVAxjSltL43aoiy+W7GqA2hlGkca5Zycigfa8QaSqlnKe06TTE71amejbzUSinSTdMT1e0D4WWakvZcZqWMfU11RtOtwsxkqprtQClL0QtenJms/3xrOnvZjmlKGfuRcEgbcWYyVc1tIzT0lOUmFr8qUko2pl+vl6ZurZT2ojRwSRkCpYTLAUoJkDHpcmx4cO/2wiFqgJpSVhjaeZUs2rAJaBmUshHKxxqx9iylo1+htspS4UA+pZmESi5VbDNLOR+bvdUYqZonU8qVYyulLLN7pVw7Jt74uu8IDT3FmFiNpYY2SKNfT09fVziQiYr77ku3Ucp1QV3WozVqCkq1nfelo18LdgoNElbj3ZcuMt/bNSpZ3CyUj51BKB87g1DJ5WYRmuZlQGnhklAdZM+hTG+zUE64MJTmbRmqwe85QkOHdXj33vWFt07siMVK6QVGP9aZZ+T2e/QC4YNk8EfEieLgxLo7BarYG27Z5Tovfp/LyH5Lg5yM+Mah3pzkNxrxZpWwQDDYIM+H129dsgFDJYubhfKxMwjlY2cQKrncLELTvAwoLVwSqoPsOZTpbRbKCReG0rwtQzX4PUdo6LAOr9/WAnWJ2Uwpe4pvkcEfkVFKWT6PXgbOIPunQS4lm8EutYp8hty2qKuulJcOlSxuFsrHziCUj51BqORyswhN8zKgtHBJqA6y51Cmt1koJ1wYSvO2DNXg9xyhoQNkrKSURRoZvH5ugQz+iFytWcoyjQZGgxyDbB5BEfObot0PO3VfTITpYj1Bl5SBXaKSxc1C+dgZhPKxMwiVXG4WoWleBpQWLgnVQfYcyvQ2C+WEC0Np3pahGvyeIzR0gIytZyn9r4nyT+pvyTiltGQzUWfzLCUNchnplwteC7O2EVSze26w5OrhG6hLdiPoVUMli5uF8rEzCOVjZxAqudwsQtO8DCgtXBKqg+w5lOltFsoJF4bSvC1DNfg9R2joABnbKqUrbDNOm6HKtLKa8QOsAQ1yRcbdDu0tNLkyKOcEAAAAgHNgwxtfXcl+ZsNNdPRJPBk8LKbRFCV9M6NBziJM5LbvXA3nItVOv5BaBQAAADgvFivlSNyUhbpTzt9z6JJOMnjYFhpkE39ra0DYY/DJds10bq9mI/1ns9tlAQAAAOBys5VSDoBSwq5AKQtKucQnQ5WOe6YXAAAAAC4RC5Syyx0HKCWgyY+gOJi7gPXw9pLSvFGTBjlIJ5ntX9YJNV+oPe56BQAAADhT1pyldHn54CxEabLC/5IH6SYsJ3zTkZpPq4HRIEdQ+v7IeWNSe7mZxxPhq7T9BCYAAAAAXEI2V0qfm+rMctxvSF5OXLX0hhN+9NLSMBaffzuSGwhFZh8nglxhfkXT4yfT8tqoNs6r1yBnkNwH25EpZZiiTAkt36+BuV8AAACA84NZyiPjjKXPpPU0TnHSRs8I9XWlU/ZOnJz/MKXmCbXHLCUAAAAAgMf6gnUHk+WOTHFNUj06Gd5eKS3arIR0nRlJnSS2UzIZh68cXyFhdsiZp/xbOU9tau6qIqaCO9oN7Oo0SAAAAAA4c/oJquiEA0ppkmGd/Z5KKSHDe2P5cTKvMZkHprdc+gbhTmdavrVmAAAAAAC4khhNCCphp52CIKCUlxchhAb1OJl62ZG6YnyZqmbdPAEAAAAA4IpipDHOM3llCLSVUs9OoZR7IZ08PKJS6k8BAAAAAMCVwyiG9Avjiu4mR+sOVaW09nhxoVTCT1mNUwyU8oh4adxQKfuXAAAAAABw5egcMtDd0dpQyvCWdRM5Ucks5U5QSqkmLe38spFFccodqSuKO1rT8kXzRCkBAAAAAK4uE5XSSkQy49W/i1LuhdQhu+lj74FVCfTq6E+tlNJEUP2a42lWhgkAAAAAAFcOowzyLsjOMItKaQ1C+qchWiVKuRfEHKPDz0AKgm2meul1MZLOWApiC9AbAgAAAACAq4dRiX5CK7WP0ixlHZRyN+STh+K8xpOUzVgKq0y+ORBWKb9+COVj6wEAAAAAgCuIMYugFXbKMQhC7cbXQLgdUpC4xhAo5XFxd7EW7m5dF98I1LQ1AAAAAABcOeIsVD+z1VBKN+Olp6aKC6uglEfGndFJlj8dd8pHT0wDAAAAAMBVoq6U1lZKP8hSW14CpTw6bqLymD+c4+56ZYoSAAAAAABKDM1Sjpq6rIJSAgAAAAAAnDENpbTwLCUAAAAAAACcBpQSAAAAAAAAZoJSAgAAAAAAwExQSgAAAAAAAJgJSgkAAAAAAAAzQSkBAAAAAABgJiglAAAAAAAAzASlBAAAAAAAgJmglAAAAAAAADATlBIAAAAAAABmglICAAAAAADATFBKAAAAAAAAmAlKCQAAAAAAALP427/9/wHCR300nSTgAAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='img/captum_result.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 읽을 거리\n",
    "\n",
    "- [**Captum**](https://github.com/pytorch/captum) 라이브러리에서 사용한 **_IntegratedGradients_** 기법을 더 자세히 이해하기 위해서는 [**논문**](https://arxiv.org/abs/1703.01365)을 참조해주세요.\n",
    "- 감정 분류 모델과 관련해 추가 학습을 원하시는 분들은 Ben Trevett이 작성한 [**pytorch-sentiment-analysis**](https://github.com/bentrevett/pytorch-sentiment-analysis) 튜토리얼을 참조해주세요. (**강추!**)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}