transformer-aihub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Annotated Transformer 보다 친절한 트랜스포머 튜토리얼"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 데이터 가공\n",
    "- 먼저, 튜토리얼에 사용하기 위해 [**AI Hub**](http://www.aihub.or.kr/)에서 [**한국어-영어 번역 말뭉치**](http://www.aihub.or.kr/aidata/87) 데이터 다운로드를 요청합니다.\n",
    "- 다운로드 요청 후, 약 **2일** 내에 승인 결과가 메일로 전달된다고 합니다.\n",
    "- 말뭉치는 기본적으로 **엑셀 파일**로 제공되지만, 실험의 편의를 위해 **CSV** 파일로 변환해 사용하도록 합니다.\n",
    "\n",
    "_cf. 현재 AI Hub에서는 다양한 한영 번역 데이터셋을 구축해 총 **160만 쌍**의 데이터를 제공해주고 있지만, 모든 문장을 훈련시키기에는 데이터가 과도하므로 본 튜토리얼에서는 **구어체 데이터 1 & 2**만을 사용하도록 합니다._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 두 개 코퍼스 파일 CSV 포맷으로 변환 저장\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "xls_a = pd.read_excel('data/spoken1.xlsx', index_col=None)\n",
    "xls_b = pd.read_excel('data/spoken2.xlsx', index_col=None)\n",
    "\n",
    "xls_a.to_csv('data/spoken1.csv', encoding='utf-8', index=False)\n",
    "xls_b.to_csv('data/spoken2.csv', encoding='utf-8', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 이제, **CSV**로 변환한 말뭉치 파일을 `pandas` 라이브러리를 이용해 읽어옵니다.\n",
    "- 읽어온 파일의 각 행을 돌며, **한국어 문장**과 **영어 문장**을 각각의 리스트에 저장해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CSV로 변환한 말뭉치 파일 로드 및 합병\n",
    "\n",
    "data_a = pd.read_csv('data/spoken1.csv', encoding='utf-8')\n",
    "data_b = pd.read_csv('data/spoken2.csv', encoding='utf-8')\n",
    "\n",
    "data = pd.concat([data_b, data_a], ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SID</th>\n",
       "      <th>원문</th>\n",
       "      <th>번역문</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>200001</td>\n",
       "      <td>0 설정을 입력하고 안정될 때까지 5분 동안 기다린 후 OK 버튼을 길게 누르십시오.</td>\n",
       "      <td>Enter into 0 setting, and wait for 5 minutes t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>200002</td>\n",
       "      <td>0은 그들에게 아무것도 아니었지만 무는 숫자일 수가 없습니다.</td>\n",
       "      <td>The zero was nothing for them but nothing coul...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>200003</td>\n",
       "      <td>1,015버전에서 핫키 버그가 있습니다.</td>\n",
       "      <td>There is a Hotkey bug in the 1,015 version.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>200004</td>\n",
       "      <td>1,390점에서 1,440점을 득점한 사람은 재판을 위해 걸러집니다.</td>\n",
       "      <td>Individuals who got a score between 1,390 and ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>200005</td>\n",
       "      <td>1,400년보다 오래 전의 유적지에 있는 최초의 성당에서 숭배자들은 그것을 인지했을...</td>\n",
       "      <td>Indeed, worshippers at the very first cathedra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>399995</td>\n",
       "      <td>199996</td>\n",
       "      <td>나는 먼저 청소기로 바닥을 밀었어요.</td>\n",
       "      <td>First of all, I vacuumed the floor.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>399996</td>\n",
       "      <td>199997</td>\n",
       "      <td>나는 먼저 팀 과제를 하고 놀러 갔어요.</td>\n",
       "      <td>I did the team assignment first and went out t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>399997</td>\n",
       "      <td>199998</td>\n",
       "      <td>나는 비 같은 멋진 연예인을 좋아해요.</td>\n",
       "      <td>I like cool entertainer like Rain.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>399998</td>\n",
       "      <td>199999</td>\n",
       "      <td>나는 멋진 자연 경치를 보고 눈물을 흘렸어.</td>\n",
       "      <td>I cried seeing the amazing scenery.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>399999</td>\n",
       "      <td>200000</td>\n",
       "      <td>나는 멋진 중학교 생활을 기대합니다.</td>\n",
       "      <td>I look forward to a great middle school experi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>400000 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           SID                                                 원문  \\\n",
       "0       200001    0 설정을 입력하고 안정될 때까지 5분 동안 기다린 후 OK 버튼을 길게 누르십시오.   \n",
       "1       200002                 0은 그들에게 아무것도 아니었지만 무는 숫자일 수가 없습니다.   \n",
       "2       200003                             1,015버전에서 핫키 버그가 있습니다.   \n",
       "3       200004             1,390점에서 1,440점을 득점한 사람은 재판을 위해 걸러집니다.   \n",
       "4       200005  1,400년보다 오래 전의 유적지에 있는 최초의 성당에서 숭배자들은 그것을 인지했을...   \n",
       "...        ...                                                ...   \n",
       "399995  199996                               나는 먼저 청소기로 바닥을 밀었어요.   \n",
       "399996  199997                             나는 먼저 팀 과제를 하고 놀러 갔어요.   \n",
       "399997  199998                              나는 비 같은 멋진 연예인을 좋아해요.   \n",
       "399998  199999                           나는 멋진 자연 경치를 보고 눈물을 흘렸어.   \n",
       "399999  200000                               나는 멋진 중학교 생활을 기대합니다.   \n",
       "\n",
       "                                                      번역문  \n",
       "0       Enter into 0 setting, and wait for 5 minutes t...  \n",
       "1       The zero was nothing for them but nothing coul...  \n",
       "2             There is a Hotkey bug in the 1,015 version.  \n",
       "3       Individuals who got a score between 1,390 and ...  \n",
       "4       Indeed, worshippers at the very first cathedra...  \n",
       "...                                                   ...  \n",
       "399995                First of all, I vacuumed the floor.  \n",
       "399996  I did the team assignment first and went out t...  \n",
       "399997                 I like cool entertainer like Rain.  \n",
       "399998                I cried seeing the amazing scenery.  \n",
       "399999  I look forward to a great middle school experi...  \n",
       "\n",
       "[400000 rows x 3 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 합병된 데이터 프레임 확인\n",
    "\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 두 코퍼스를 병합해 총 **400,000개**의 **병렬 문장 쌍**이 만들어진 것을 확인하였습니다!\n",
    "- 이제 해당 데이터 프레임을 돌며, 한국어 문장과 영어 문장을 리스트에 각각 저장해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 한국어, 영어 데이터를 별개 리스트에 저장\n",
    "\n",
    "kor_lines = []\n",
    "eng_lines = []\n",
    "\n",
    "for _, row in data.iterrows():\n",
    "    _, kor, eng = row\n",
    "    kor_lines.append(kor)\n",
    "    eng_lines.append(eng)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**언어별 데이터**가 잘 저장되었는지 일부 데이터를 출력해 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[KOR]: 0 설정을 입력하고 안정될 때까지 5분 동안 기다린 후 OK 버튼을 길게 누르십시오.\n",
      "[ENG]: Enter into 0 setting, and wait for 5 minutes to make it stable, then long-press OK button.\n",
      "\n",
      "[KOR]: 0은 그들에게 아무것도 아니었지만 무는 숫자일 수가 없습니다.\n",
      "[ENG]: The zero was nothing for them but nothing couldn't be a number.\n",
      "\n",
      "[KOR]: 1,015버전에서 핫키 버그가 있습니다.\n",
      "[ENG]: There is a Hotkey bug in the 1,015 version.\n",
      "\n",
      "[KOR]: 1,390점에서 1,440점을 득점한 사람은 재판을 위해 걸러집니다.\n",
      "[ENG]: Individuals who got a score between 1,390 and 1,440 are selected for a judge.\n",
      "\n",
      "[KOR]: 1,400년보다 오래 전의 유적지에 있는 최초의 성당에서 숭배자들은 그것을 인지했을 것입니다.\n",
      "[ENG]: Indeed, worshippers at the very first cathedral on this site, over 1,400 years ago, would have still recognized it.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for kor, eng in zip(kor_lines[:5], eng_lines[:5]):\n",
    "    print(f'[KOR]: {kor}')\n",
    "    print(f'[ENG]: {eng}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tokenizers** 라이브러리를 학습시키기 위한 **훈련용 텍스트 파일**을 만들어줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 한국어 토크나이저 훈련 데이터 제작\n",
    "\n",
    "with open('train_korean.txt', 'w', encoding='utf-8') as f:\n",
    "    for line in kor_lines:\n",
    "        print(line, file=f)\n",
    "\n",
    "\n",
    "# 영어 토크나이저 훈련 데이터 제작\n",
    "\n",
    "with open('train_english.txt', 'w', encoding='utf-8') as f:\n",
    "    for line in eng_lines:\n",
    "        print(line, file=f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. BPE 토크나이저 학습\n",
    "- 앞서 가공한 데이터들을 활용해 **BPE 토크나이저**를 학습시킵니다.\n",
    "- 토크나이저를 학습시키기 앞서 프로젝트 전반에 사용될 변수 사전을 정의합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {\n",
    "    'batch_size': 64,\n",
    "    'num_epoch': 15,\n",
    "    'dropout': 0.1,\n",
    "    'min_frequency': 3,\n",
    "    \n",
    "    'vocab_size': 20000,\n",
    "    'num_layers': 6,\n",
    "    'num_heads': 8,\n",
    "    'hidden_dim': 512,\n",
    "    'ffn_dim': 2048,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 한국어와 영어 토크나이저를 **별도로 초기화해 훈련**시킵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers import BPETokenizer\n",
    "\n",
    "# 한국어 토크나이저 초기화\n",
    "\n",
    "kor_tokenizer = BPETokenizer()\n",
    "\n",
    "\n",
    "# 한국어 토크나이저 훈련\n",
    "\n",
    "kor_tokenizer.train(\n",
    "    ['train_korean.txt'],\n",
    "    vocab_size=params['vocab_size'],\n",
    "    min_frequency=params['min_frequency'],\n",
    "    special_tokens=['[PAD]', '[SOS]', '[EOS]', '[UNK]'],\n",
    "    suffix=''\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 영어 토크나이저 초기화\n",
    "\n",
    "eng_tokenizer = BPETokenizer()\n",
    "\n",
    "\n",
    "# 영어 토크나이저 훈련\n",
    "\n",
    "eng_tokenizer.train(\n",
    "    ['train_english.txt'],\n",
    "    vocab_size=params['vocab_size'],\n",
    "    min_frequency=params['min_frequency'],\n",
    "    special_tokens=['[PAD]', '[SOS]', '[EOS]', '[UNK]'],\n",
    "    suffix=''\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**패딩 옵션**과 **후처리 작업** 등에 사용될 **스페셜 토큰**들의 아이디를 저장해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pad_idx = kor_tokenizer.token_to_id('[PAD]')\n",
    "sos_idx = kor_tokenizer.token_to_id('[SOS]')\n",
    "eos_idx = kor_tokenizer.token_to_id('[EOS]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 훈련된 토크나이저로 토큰화 진행"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizers의 `encode_batch` 함수를 활용해 각 데이터들에 대해 **토큰화 작업을 수행**해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "kor_encoded_data = kor_tokenizer.encode_batch(kor_lines)\n",
    "eng_encoded_data = eng_tokenizer.encode_batch(eng_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "토큰화 작업이 잘 수행되었는지 기존 데이터와 비교해 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Orig]: 0 설정을 입력하고 안정될 때까지 5분 동안 기다린 후 OK 버튼을 길게 누르십시오.\n",
      "[Proc]: ['0', '설', '정을', '입력', '하고', '안정', '될', '때까지', '5분', '동안', '기다', '린', '후', 'ok', '버튼을', '길게', '누르', '십시오', '.']\n",
      "\n",
      "[Orig]: 0은 그들에게 아무것도 아니었지만 무는 숫자일 수가 없습니다.\n",
      "[Proc]: ['0', '은', '그들에게', '아무것도', '아니', '었지만', '무', '는', '숫자', '일', '수가', '없습니다', '.']\n",
      "\n",
      "[Orig]: 1,015버전에서 핫키 버그가 있습니다.\n",
      "[Proc]: ['1', ',', '0', '15', '버전', '에서', '키', '버', '그가', '있습니다', '.']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 한국어 데이터 토큰화 작업 결과 출력\n",
    "\n",
    "for origin, processed in zip(kor_lines[:3], kor_encoded_data[:3]):\n",
    "    print(f'[Orig]: {origin}')\n",
    "    print(f'[Proc]: {processed.tokens}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Orig]: Enter into 0 setting, and wait for 5 minutes to make it stable, then long-press OK button.\n",
      "[Proc]: ['enter', 'into', '0', 'setting', ',', 'and', 'wait', 'for', '5', 'minutes', 'to', 'make', 'it', 'stable', ',', 'then', 'long', '-', 'press', 'ok', 'button', '.']\n",
      "\n",
      "[Orig]: The zero was nothing for them but nothing couldn't be a number.\n",
      "[Proc]: ['the', 'zero', 'was', 'nothing', 'for', 'them', 'but', 'nothing', 'couldn', \"'\", 't', 'be', 'a', 'number', '.']\n",
      "\n",
      "[Orig]: There is a Hotkey bug in the 1,015 version.\n",
      "[Proc]: ['there', 'is', 'a', 'hot', 'key', 'bug', 'in', 'the', '1', ',', '0', '15', 'version', '.']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 영어 데이터 토큰화 작업 결과 출력\n",
    "\n",
    "for origin, processed in zip(eng_lines[:3], eng_encoded_data[:3]):\n",
    "    print(f'[Orig]: {origin}')\n",
    "    print(f'[Proc]: {processed.tokens}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 토큰화 결과에 후처리 로직 적용"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 이제 토큰화 작업이 수행된 결과에 **[PAD]** 토큰을 붙여줄 차례입니다.\n",
    "- **[PAD]** 토큰은 모델이 입력으로 받는 **최대 길이** 보다 길이가 짧은 문장들에 한해 부여되는 토큰이므로, \n",
    "- **최대 길이**로 설정할 적정 길이를 찾기 위해 각 언어 쌍의 평균 길이와 최대 길이를 계산합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11.27639, 49)"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 한국어 데이터 평균 및 최대 길이 계산\n",
    "\n",
    "kor_len_max = max(len(line.tokens) for line in kor_encoded_data)\n",
    "kor_len = 0\n",
    "\n",
    "for line in kor_encoded_data:\n",
    "    kor_len += len(line.tokens)\n",
    "kor_len_avg = kor_len / len(kor_encoded_data)\n",
    "\n",
    "kor_len_avg, kor_len_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12.8836825, 65)"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 영어 데이터 평균 및 최대 길이 계산\n",
    "\n",
    "eng_len_max = max(len(line.tokens) for line in eng_encoded_data)\n",
    "eng_len = 0\n",
    "\n",
    "for line in eng_encoded_data:\n",
    "    eng_len += len(line.tokens)\n",
    "eng_len_avg = eng_len / len(eng_encoded_data)\n",
    "\n",
    "eng_len_avg, eng_len_max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이터셋 내 문장들이 그렇게 긴 편이 아니므로 **32**로 입력 값의 **최대 길이**로 정해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "params['max_len'] = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막으로 **[PAD]** 토큰을 붙여주는 `pad_sentence` 함수와 \n",
    "\n",
    "문장의 시작과 끝을 알리는 **[SOS]**, **[EOS]** 토큰을 붙여주는 후처리 함수 `postprocess`를 정의해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_sentence(input_ids):\n",
    "    '''최대 길이보다 짧은 문장들에 [PAD] 토큰 부여'''\n",
    "\n",
    "    num_pad = params['max_len'] - len(input_ids)\n",
    "    input_ids.extend([pad_idx] * num_pad)\n",
    "    return input_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def postprocess(input_ids):\n",
    "    '''입력 문장에 [SOS] 토큰과 [EOS] 토큰 부여'''\n",
    "    \n",
    "    input_ids = pad_sentence(input_ids)\n",
    "    \n",
    "    input_ids = [sos_idx] + input_ids\n",
    "    \n",
    "    input_ids = input_ids[:params['max_len']]\n",
    "\n",
    "    if pad_idx in input_ids:\n",
    "        pad_start = input_ids.index(pad_idx)\n",
    "        input_ids[pad_start] = eos_idx\n",
    "    else:\n",
    "        input_ids[-1] = eos_idx\n",
    "    \n",
    "    return input_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "앞서 정의한 두 함수를 이용하면 결과 값이 다음과 같이 바뀝니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "토큰화 결과: ['우리', '진짜', '별', '나', '대', '그냥', '내가', '너무', '좋아해', '넌', '그걸', '너무', '잘', '알고', '날', '쥐', '락', '펴', '락', '해', '나도', '마찬가지', '인', '걸']\n"
     ]
    }
   ],
   "source": [
    "# 기본 토큰화 작업 결과 \n",
    "\n",
    "sent = '우리 진짜 별나대 그냥 내가 너무 좋아해 넌 그걸 너무 잘 알고 날 쥐락펴락해 나도 마찬가지인걸'\n",
    "\n",
    "proc_sent = kor_tokenizer.encode(sent)\n",
    "print(f'토큰화 결과: {proc_sent.tokens}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "후처리 결과: [1, 1096, 3879, 472, 173, 249, 2811, 1190, 1393, 7045, 196, 4602, 1393, 717, 1833, 177, 758, 316, 917, 316, 950, 2186, 7015, 703, 78, 2, 0, 0, 0, 0, 0, 0]\n",
      "\n",
      "후처리 해석: [SOS]우리진짜별나대그냥내가너무좋아해넌그걸너무잘알고날쥐락펴락해나도마찬가지인걸[EOS][PAD][PAD][PAD][PAD][PAD][PAD]\n"
     ]
    }
   ],
   "source": [
    "# 토큰화 + 후처리 작업 결과\n",
    "\n",
    "post_proc_sent = postprocess(proc_sent.ids)\n",
    "\n",
    "print(f'후처리 결과: {post_proc_sent}\\n')\n",
    "print(f'후처리 해석: {kor_tokenizer.decode(post_proc_sent)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 후처리 함수를 활용해 모든 데이터셋들에 대해 **후처리 작업을 진행**해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "kor_processed_data = [postprocess(data.ids) for data in kor_encoded_data]\n",
    "eng_processed_data = [postprocess(data.ids) for data in eng_encoded_data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 모든 데이터셋을 텐서형 데이터로 변환"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 전처리와 후처리를 모두 마친 데이터들을 `torch.Tensor`로 변환해줍니다.\n",
    "- 변환 후, `DataLoader`를 활용해 데이터들을 **배치**로 만들어줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "torch.backends.cudnn.deterministic = True\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "kor_tensors = [torch.LongTensor(line).to(device) for line in kor_processed_data]\n",
    "eng_tensors = [torch.LongTensor(line).to(device) for line in eng_processed_data]\n",
    "\n",
    "src_iter = data.DataLoader(kor_tensors, batch_size=params['batch_size'])\n",
    "tgt_iter = data.DataLoader(eng_tensors, batch_size=params['batch_size'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "배치 데이터가 잘 생성되었는지 출력을 통해 확인합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Transformer 모델링"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 먼저 모델 구현에 필요한 라이브러리들을 모두 임포트합니다.\n",
    "- 실험을 함에 있어 항상 실험의 **Reproducibility**를 보장하기 위해 Seed 설정을 해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "torch.manual_seed(32)\n",
    "torch.cuda.manual_seed(32)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5-1. (Masked) Multi-Head Attention 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Sub Layers\n",
    "### Multi-Head Attention\n",
    "\n",
    "![](img/mha.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    '''멀티 헤드 어텐션 레이어'''\n",
    "    def __init__(self, params):\n",
    "        super(MultiHeadAttention, self).__init__()\n",
    "        assert params['hidden_dim'] % params['num_heads'] == 0, \"hidden dimension must be divisible by the number of heads\"\n",
    "        self.num_heads = params['num_heads']\n",
    "        self.attn_dim = params['hidden_dim'] // self.num_heads\n",
    "        \n",
    "        self.q_w = nn.Linear(params['hidden_dim'], self.num_heads * self.attn_dim)\n",
    "        self.k_w = nn.Linear(params['hidden_dim'], self.num_heads * self.attn_dim)\n",
    "        self.v_w = nn.Linear(params['hidden_dim'], self.num_heads * self.attn_dim)\n",
    "        \n",
    "        self.o_w = nn.Linear(self.num_heads * self.attn_dim, params['hidden_dim'])\n",
    "        \n",
    "    def forward(self, q, k, v, mask=None):\n",
    "        \" q, k, v = [배치 사이즈, 문장 길이, 은닉 차원] \"\n",
    "        \n",
    "        batch_size = q.size(0)\n",
    "        \n",
    "        q = self.q_w(q).view(batch_size, -1, self.num_heads, self.attn_dim).transpose(1, 2)\n",
    "        k = self.k_w(k).view(batch_size, -1, self.num_heads, self.attn_dim).transpose(1, 2)\n",
    "        v = self.v_w(v).view(batch_size, -1, self.num_heads, self.attn_dim).transpose(1, 2)\n",
    "        # q, k, v = [배치 사이즈, 헤드 갯수, 문장 길이, 어텐션 차원]\n",
    "        \n",
    "        attn = torch.matmul(q, k.transpose(-1, -2))\n",
    "        # attn = [배치 사이즈, 헤드 갯수, 문장 길이, 문장 길이]\n",
    "        \n",
    "        if mask is not None:\n",
    "            mask = mask.unsqueeze(1)\n",
    "            attn.masked_fill(mask==0, -1e9)\n",
    "        \n",
    "        score = F.softmax(attn, dim=-1)\n",
    "        # score = [배치 사이즈, 헤드 갯수, 문장 길이, 문장 길이]\n",
    "        \n",
    "        output = torch.matmul(score, v)\n",
    "        # output = [배치 사이즈, 헤드 갯수, 문장 길이, 어텐션 차원]\n",
    "        \n",
    "        output = output.transpose(1, 2).contiguous()\n",
    "        # output = [배치 사이즈, 문장 길이, 헤드 갯수, 어텐션 차원]\n",
    "        \n",
    "        output = output.view(batch_size, -1, self.num_heads * self.attn_dim)\n",
    "        # output = [배치 사이즈, 문장 길이, 은닉 차원]\n",
    "        \n",
    "        output = self.o_w(output)\n",
    "        # output = [배치 사이즈, 문장 길이, 은닉 차원]\n",
    "        \n",
    "        return output, score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Masked Multi Head Attention을 위한 Subsequent Mask를 생성해주어야 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_subsequent_mask(tgt):\n",
    "    batch_size, tgt_len = tgt.size()\n",
    "    \n",
    "    subsequent_mask = torch.triu(torch.ones(tgt_len, tgt_len), diagonal=1).bool()\n",
    "    # subsequent_mask = [타겟 문장 길이, 타겟 문장 길이]\n",
    "    \n",
    "    subsequent_mask = subsequent_mask.unsqueeze(0).repeat(batch_size, 1, 1).to(device)\n",
    "    # subsquent_mask = [배치 사이즈, 타겟 문장 길이, 타겟 문장 길이]\n",
    "    \n",
    "    return subsequent_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "사용 예는 아래와 같습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['왜', '들', '그리', '다운', '돼', '있어', '?', '뭐가', '문제', '야', 'say', 'something', '분위기가', '겁', '나', '싸', '해']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f71386cf2d0>"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATEAAAEvCAYAAAAtufaDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAPCElEQVR4nO3df4xldXnH8fenu8uugxqgiD9YUsAgLSVUzGhVWmtZaVcl4B/9A1KbbSWZpGkVjRYhJDX9z1Tjj6RGs4EVUsgSg6jEqEBQS5oodVkBFxdlqwgDq4shVaMJLOHpH3NJhtm7zJl7z5ydL7xfyWTuPffMfZ6dvfvZ7zn3zDOpKiSpVb93pBuQpGkYYpKaZohJapohJqlphpikphlikpq2fshiR2VjbeLoib/+NWf9rsduJLXkrnuf+GVVvWzp9kFDbBNH86fZMvHX33LL3T12I6kl616572fjtns4Kalphpikpk0VYkm2JvlRkn1JLu+rKUnqauIQS7IO+AzwduAM4OIkZ/TVmCR1Mc1K7A3Avqr6SVU9CdwAXNhPW5LUzTQhdiLw8KL786NtkjSYaS6xyJhth8z1STIHzAFsYmaKcpJ0qGlWYvPASYvubwYeXbpTVW2vqtmqmt3AxinKSdKhpgmx7wGnJTklyVHARcDN/bQlSd1MfDhZVU8l+WfgFmAdsKOq7uutM0nqYKofO6qqrwFf66kXSVoxr9iX1DRDTFLTDDFJTRt0FM+0/vpVr53q62951FE+0vONKzFJTTPEJDXNEJPUNENMUtMMMUlNM8QkNc0Qk9Q0Q0xS0wwxSU0zxCQ1zRCT1DRDTFLTDDFJTTPEJDXNEJPUtKbmiU3LeWTS848rMUlNM8QkNc0Qk9S0iUMsyUlJvpVkb5L7klzaZ2OS1MU0J/afAj5YVbuTvAS4K8ltVfXDnnqTpGVNvBKrqv1VtXt0+zfAXuDEvhqTpC56OSeW5GTgbODOPp5Pkrqa+jqxJC8Gvgi8v6p+PebxOWAOYBMz05aTpGeZaiWWZAMLAXZ9Vd00bp+q2l5Vs1U1u4GN05STpENM8+5kgKuBvVX1if5akqTuplmJnQP8HXBukrtHH+/oqS9J6mTic2JV9d9AeuxFklbMK/YlNc0Qk9Q0Q0xS015Q88Sm5Twyae1xJSapaYaYpKYZYpKaZohJapohJqlphpikphlikppmiElqmiEmqWmGmKSmGWKSmmaISWqaISapaYaYpKYZYpKa5jyxATmPTOqfKzFJTTPEJDXNEJPUtKlDLMm6JN9P8tU+GpKklehjJXYpsLeH55GkFZsqxJJsBt4JXNVPO5K0MtOuxD4FXAY83UMvkrRiE4dYkvOBA1V11zL7zSXZlWTXQZ6YtJwkjTXNSuwc4IIkDwI3AOcmuW7pTlW1vapmq2p2AxunKCdJh5o4xKrqiqraXFUnAxcB36yqd/fWmSR14HVikprWy89OVtW3gW/38VyStBKuxCQ1zRCT1DRDTFLTnCfWEOeRSYdyJSapaYaYpKYZYpKaZohJapohJqlphpikphlikppmiElqmiEmqWmGmKSmGWKSmmaISWqaISapaYaYpKYZYpKa5jyxFxDnken5yJWYpKYZYpKaZohJatpUIZbkmCQ3Jrk/yd4kb+qrMUnqYtoT+58GvlFVf5PkKGCmh54kqbOJQyzJS4G3AH8PUFVPAk/205YkdTPN4eSpwGPA55N8P8lVSY7uqS9J6mSaEFsPvA74bFWdDfwWuHzpTknmkuxKsusgT0xRTpIONU2IzQPzVXXn6P6NLITas1TV9qqararZDWycopwkHWriEKuqnwMPJzl9tGkL8MNeupKkjqZ9d/K9wPWjdyZ/AvzD9C1JUndThVhV3Q3M9tSLJK2YV+xLapohJqlphpikpjlPTJ05j0xrkSsxSU0zxCQ1zRCT1DRDTFLTDDFJTTPEJDXNEJPUNENMUtMMMUlNM8QkNc0Qk9Q0Q0xS0wwxSU0zxCQ1zRCT1DTniWkwziPTanAlJqlphpikphlikpo2VYgl+UCS+5LsSbIzyaa+GpOkLiYOsSQnAu8DZqvqTGAdcFFfjUlSF9MeTq4HXpRkPTADPDp9S5LU3cQhVlWPAB8HHgL2A7+qqlv7akySupjmcPJY4ELgFOBVwNFJ3j1mv7kku5LsOsgTk3cqSWNMczj5NuCnVfVYVR0EbgLevHSnqtpeVbNVNbuBjVOUk6RDTRNiDwFvTDKTJMAWYG8/bUlSN9OcE7sTuBHYDfxg9Fzbe+pLkjqZ6mcnq+ojwEd66kWSVswr9iU1zRCT1DRDTFLTnCemZjiPTOO4EpPUNENMUtMMMUlNM8QkNc0Qk9Q0Q0xS0wwxSU0zxCQ1zRCT1DRDTFLTDDFJTTPEJDXNEJPUNENMUtMMMUlNc56YXjCcR/b85EpMUtMMMUlNM8QkNW3ZEEuyI8mBJHsWbTsuyW1JHhh9PnZ125Sk8bqsxK4Bti7Zdjlwe1WdBtw+ui9Jg1s2xKrqDuDxJZsvBK4d3b4WeFfPfUlSJ5OeE3t5Ve0HGH0+ob+WJKm7Vb9OLMkcMAewiZnVLifpBWbSldgvkrwSYPT5wOF2rKrtVTVbVbMb2DhhOUkab9IQuxnYNrq9DfhKP+1I0sp0ucRiJ/Ad4PQk80kuAT4KnJfkAeC80X1JGtyy58Sq6uLDPLSl514kacW8Yl9S0wwxSU0zxCQ1zXliUkfOI1ubXIlJapohJqlphpikphlikppmiElqmiEmqWmGmKSmGWKSmmaISWqaISapaYaYpKYZYpKaZohJapohJqlphpikpjlPTBqI88hWhysxSU0zxCQ1zRCT1LQuvzx3R5IDSfYs2vaxJPcnuTfJl5Ics7ptStJ4XVZi1wBbl2y7DTizqs4Cfgxc0XNfktTJsiFWVXcAjy/ZdmtVPTW6+11g8yr0JknL6uOc2HuAr/fwPJK0YlNdJ5bkSuAp4Prn2GcOmAPYxMw05STpEBOHWJJtwPnAlqqqw+1XVduB7QAvzXGH3U+SJjFRiCXZCnwY+Iuq+l2/LUlSd10usdgJfAc4Pcl8kkuA/wBeAtyW5O4kn1vlPiVprGVXYlV18ZjNV69CL5K0Yl6xL6lphpikphlikprmPDGpEc4jG8+VmKSmGWKSmmaISWqaISapaYaYpKYZYpKaZohJapohJqlphpikphlikppmiElqmiEmqWmGmKSmGWKSmmaISWqa88SkF4jn6zwyV2KSmmaISWqaISapaV1+ee6OJAeS7Bnz2IeSVJLjV6c9SXpuXVZi1wBbl25MchJwHvBQzz1JUmfLhlhV3QE8PuahTwKXAdV3U5LU1UTnxJJcADxSVff03I8krciKrxNLMgNcCfxVx/3ngDmATcystJwkPadJVmKvBk4B7knyILAZ2J3kFeN2rqrtVTVbVbMb2Dh5p5I0xopXYlX1A+CEZ+6Pgmy2qn7ZY1+S1EmXSyx2At8BTk8yn+SS1W9LkrpZdiVWVRcv8/jJvXUjSSvkFfuSmmaISWqaISapac4Tk9TJWp1H5kpMUtMMMUlNM8QkNc0Qk9Q0Q0xS0wwxSU0zxCQ1zRCT1DRDTFLTDDFJTTPEJDXNEJPUNENMUtMMMUlNM8QkNc15YpIGMe08Mtg3dqsrMUlNM8QkNc0Qk9S0Lr88d0eSA0n2LNn+3iQ/SnJfkn9fvRYl6fC6rMSuAbYu3pDkL4ELgbOq6o+Bj/ffmiQtb9kQq6o7gMeXbP5H4KNV9cRonwOr0JskLWvSc2KvAf48yZ1J/ivJ6/tsSpK6mvQ6sfXAscAbgdcDX0hyalXV0h2TzAFzAJuYmbRPSRpr0pXYPHBTLfgf4Gng+HE7VtX2qpqtqtkNbJy0T0kaa9IQ+zJwLkCS1wBHAb/sqylJ6mrZw8kkO4G3AscnmQc+AuwAdowuu3gS2DbuUFKSVtuyIVZVFx/moXf33IskrZhX7EtqmiEmqWmGmKSmZcjz8UkeA372HLscz5F9l9P61rf+2q3/B1X1sqUbBw2x5STZVVWz1re+9a3flYeTkppmiElq2loLse3Wt771rb8Sa+qcmCSt1FpbiUnSiqyJEEuydTTqel+SyweufVKSbyXZOxq1femQ9Rf1sS7J95N89QjUPibJjUnuH30f3jRw/Q+Mvvd7kuxMsmmAmoeMXU9yXJLbkjww+nzswPU/Nvo7uDfJl5IcM2T9RY99KEklGTuZZjXrTzL2/oiHWJJ1wGeAtwNnABcnOWPAFp4CPlhVf8TCfLR/Grj+My4F9h6BugCfBr5RVX8I/MmQfSQ5EXgfMFtVZwLrgIsGKH0NS8auA5cDt1fVacDto/tD1r8NOLOqzgJ+DFwxcH2SnAScBzy0irXH1p907P0RDzHgDcC+qvpJVT0J3MDCH2QQVbW/qnaPbv+GhX/AJw5VHyDJZuCdwFVD1h3VfinwFuBqgKp6sqr+b+A21gMvSrIemAEeXe2Chxm7fiFw7ej2tcC7hqxfVbdW1VOju98FNg9Zf+STwGXAqp4s73Ps/VoIsROBhxfdn2fgEHlGkpOBs4E7By79KRZeOE8PXBfgVOAx4POjw9mrkhw9VPGqeoSF/3EfAvYDv6qqW4eqv8TLq2r/qK/9wAlHqA+A9wBfH7JgkguAR6rqniHrLjLR2Pu1EGIZs23wt0yTvBj4IvD+qvr1gHXPBw5U1V1D1VxiPfA64LNVdTbwW1b3MOpZRuedLgROAV4FHJ3kBT3mKcmVLJzmuH7AmjPAlcC/DlVzjMVj7/+FhbH34/LhWdZCiM0DJy26v5kBDicWS7KBhQC7vqpuGrI2cA5wQZIHWTiUPjfJdQPWnwfmq+qZ1eeNLITaUN4G/LSqHquqg8BNwJsHrL/YL5K8EmD0efDf4pVkG3A+8LcDDxp9NQv/kdwzei1uBnYnecWAPXQee7/YWgix7wGnJTklyVEsnNS9eajio6S/GthbVZ8Yqu4zquqKqtpcVSez8Gf/ZlUNthKpqp8DDyc5fbRpC/DDoeqzcBj5xiQzo7+LLRy5NzhuBraNbm8DvjJk8SRbgQ8DF1TV74asXVU/qKoTqurk0WtxHnjd6PUxlMnG3lfVEf8A3sHCuzH/C1w5cO0/Y+Hw9V7g7tHHO47Q9+GtwFePQN3XArtG34MvA8cOXP/fgPuBPcB/AhsHqLmThXNwB1n4B3sJ8PssvCv5wOjzcQPX38fC+eFnXoefG7L+kscfBI4f+M9/FHDd6HWwGzi3y3N5xb6kpq2Fw0lJmpghJqlphpikphlikppmiElqmiEmqWmGmKSmGWKSmvb/B+KhxwP6Hk8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_sent = '왜들 그리 다운돼있어? 뭐가 문제야 say something 분위기가 겁나 싸해'\n",
    "test_tensor = kor_tokenizer.encode(test_sent)\n",
    "test_tensor = torch.LongTensor(test_tensor.ids).to(device).unsqueeze(0)\n",
    "\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.imshow(create_subsequent_mask(test_tensor).cpu()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_src_mask(src):\n",
    "    \" source = [배치 사이즈, 소스 문장 길이] \"\n",
    "\n",
    "    src_len = src.size(1)\n",
    "    \n",
    "    src_mask = (src == pad_idx)\n",
    "    # src_mask = [배치 사이즈, 소스 문장 길이]\n",
    "    \n",
    "    src_mask = src_mask.unsqueeze(1).repeat(1, src_len, 1)\n",
    "    # src_mask = [배치 사이즈, 소스 문장 길이, 소스 문장 길이]\n",
    "\n",
    "    return src_mask.to(device)\n",
    "\n",
    "\n",
    "def create_tgt_mask(src, tgt):\n",
    "    \" src = [배치 사이즈, 소스 문장 길이] \"\n",
    "    \" tgt = [배치 사이즈, 타겟 문장 길이] \"\n",
    "    \n",
    "    batch_size, tgt_len = tgt.size()\n",
    "    \n",
    "    subsequent_mask = create_subsequent_mask(tgt)\n",
    "    \n",
    "    enc_dec_mask = (src == pad_idx)\n",
    "    tgt_mask = (tgt == pad_idx)\n",
    "    # src_mask = [배치 사이즈, 소스 문장 길이]\n",
    "    # tgt_mask = [배치 사이즈, 타겟 문장 길이]\n",
    "    \n",
    "    enc_dec_mask = enc_dec_mask.unsqueeze(1).repeat(1, tgt_len, 1).to(device)\n",
    "    tgt_mask = tgt_mask.unsqueeze(1).repeat(1, tgt_len, 1).to(device)\n",
    "    # src_mask = [배치 사이즈, 타겟 문장 길이, 소스 문장 길이]\n",
    "    # tgt_mask = [배치 사이즈, 타겟 문장 길이, 타겟 문장 길이]\n",
    "\n",
    "    tgt_mask = tgt_mask | subsequent_mask\n",
    "    \n",
    "    return enc_dec_mask, tgt_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "소스 문장의 마스크 생성 예는 아래와 같습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f714e0abe10>"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATEAAAEvCAYAAAAtufaDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAALqklEQVR4nO3db4hdB5nH8e9vM2li0hVddBdNyrZCt1pkl8rgVguyNArtKnZfKLRQURHyZtUqgtR907f7QkRfiBBqVbC0LLGwpZRW8Q+LIMFpWtamoxiqpmPjNrKsigtJis++mFtIk3Fz7z0nc+8Tvx8IM/fMnXueQybfnHPuYU6qCknq6s8WPYAkDWHEJLVmxCS1ZsQktWbEJLVmxCS1trKdK7siu2o3e7dzldLS+5u//d9Fj9DCE/95+tdV9drzl29rxHazl7/Pge1cpbT0Hn/8qUWP0MKO1x3/xVbLPZyU1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1NqgiCW5JclPkhxPcvdYQ0nStOaOWJIdwBeBW4HrgTuSXD/WYJI0jSF7Ym8FjlfVs1V1BngQuG2csSRpOkMitg947pzHG5NlL5PkYJK1JGtnOT1gdZJ0oSERyxbLLviF/VV1qKpWq2p1J7sGrE6SLjQkYhvAVec83g88P2wcSZrNkIj9ELg2yTVJrgBuBx4eZyxJms7cv4qnql5M8lHgcWAHcF9VHRttMkmawqDfJ1ZVjwKPjjSLJM3MK/YltWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktTZ3xJJcleS7SdaTHEty15iDSdI0VgZ874vAp6rqaJI/B55I8q2qemak2STpoubeE6uqk1V1dPL574B1YN9Yg0nSNEY5J5bkauAG4MgYrydJ0xpyOAlAkiuBbwCfqKrfbvH1g8BBgN3sGbo6SXqZQXtiSXayGbD7q+qhrZ5TVYeqarWqVneya8jqJOkCQ96dDPBlYL2qPjfeSJI0vSF7YjcBHwBuTvLU5M8/jjSXJE1l7nNiVfV9ICPOIkkz84p9Sa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0NjliSHUmeTPLIGANJ0izG2BO7C1gf4XUkaWaDIpZkP/Bu4N5xxpGk2QzdE/s88GngDyPMIkkzmztiSd4DvFBVT1zkeQeTrCVZO8vpeVcnSVsasid2E/DeJD8HHgRuTvL1859UVYeqarWqVneya8DqJOlCc0esqj5TVfur6mrgduA7VXXnaJNJ0hS8TkxSaytjvEhVfQ/43hivJUmzcE9MUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrgyKW5FVJDif5cZL1JG8bazBJmsbKwO//AvBYVb0vyRXAnhFmkqSpzR2xJK8E3gF8CKCqzgBnxhlLkqYz5HDyDcAp4CtJnkxyb5K9I80lSVMZErEV4C3Al6rqBuD3wN3nPynJwSRrSdbOcnrA6iTpQkMitgFsVNWRyePDbEbtZarqUFWtVtXqTnYNWJ0kXWjuiFXVr4Dnklw3WXQAeGaUqSRpSkPfnfwYcP/knclngQ8PH0mSpjcoYlX1FLA60iySNDOv2JfUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPUmhGT1JoRk9SaEZPU2qCIJflkkmNJnk7yQJLdYw0mSdOYO2JJ9gEfB1ar6s3ADuD2sQaTpGkMPZxcAV6RZAXYAzw/fCRJmt7cEauqXwKfBU4AJ4HfVNU3xxpMkqYx5HDy1cBtwDXA64G9Se7c4nkHk6wlWTvL6fknlaQtDDmcfCfws6o6VVVngYeAt5//pKo6VFWrVbW6k10DVidJFxoSsRPAjUn2JAlwAFgfZyxJms6Qc2JHgMPAUeBHk9c6NNJckjSVlSHfXFX3APeMNIskzcwr9iW1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS1ZsQktXbRiCW5L8kLSZ4+Z9lfJPlWkp9OPr760o4pSVubZk/sq8At5y27G/h2VV0LfHvyWJK23UUjVlX/Afz3eYtvA742+fxrwD+NPJckTWXec2J/VVUnASYf//KPPTHJwSRrSdbOcnrO1UnS1i75if2qOlRVq1W1upNdl3p1kv7EzBux/0ryOoDJxxfGG0mSpjdvxB4GPjj5/IPAv48zjiTNZppLLB4AfgBcl2QjyUeAfwXeleSnwLsmjyVp261c7AlVdccf+dKBkWeRpJl5xb6k1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1lJV27ey5BTwixm+5TXAry/RONvpctgOt2F5XA7bMc82/HVVvfb8hdsasVklWauq1UXPMdTlsB1uw/K4HLZjzG3wcFJSa0ZMUmvLHrFDix5gJJfDdrgNy+Ny2I7RtmGpz4lJ0sUs+56YJP2/ljZiSW5J8pMkx5O0u5tSkquSfDfJepJjSe5a9EzzSrIjyZNJHln0LPNK8qokh5P8ePJ38rZFzzSrJJ+c/Cw9neSBJLsXPdM0LvVtH5cyYkl2AF8EbgWuB+5Icv1ip5rZi8CnqupNwI3APzfchpfcBawveoiBvgA8VlVvBP6OZtuTZB/wcWC1qt4M7ABuX+xUU/sql/C2j0sZMeCtwPGqeraqzgAPsnmbuDaq6mRVHZ18/js2/9HsW+xUs0uyH3g3cO+iZ5lXklcC7wC+DFBVZ6rqfxY71VxWgFckWQH2AM8veJ6pXOrbPi5rxPYBz53zeIOGAXhJkquBG4Aji51kLp8HPg38YdGDDPAG4BTwlclh8b1J9i56qFlU1S+BzwIngJPAb6rqm4udapCpb/t4McsasWyxrOXbqEmuBL4BfKKqfrvoeWaR5D3AC1X1xKJnGWgFeAvwpaq6Afg9ze5aPzlndBtwDfB6YG+SOxc71XJY1ohtAFed83g/TXadz5VkJ5sBu7+qHlr0PHO4CXhvkp+zeUh/c5KvL3akuWwAG1X10p7wYTaj1sk7gZ9V1amqOgs8BLx9wTMNMdptH5c1Yj8Erk1yTZIr2DyB+fCCZ5pJkrB5Dma9qj636HnmUVWfqar9VXU1m38H36mqdv/7V9WvgOeSXDdZdAB4ZoEjzeMEcGOSPZOfrQM0e3PiPKPd9vGidztahKp6MclHgcfZfBfmvqo6tuCxZnUT8AHgR0memiz7l6p6dIEz/Sn7GHD/5D/FZ4EPL3iemVTVkSSHgaNsvvP9JE2u3J/c9vEfgNck2QDuYfM2j/82uQXkCeD9c7++V+xL6mxZDyclaSpGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJr/wctql9WTTGqdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "src = torch.tensor([[1, 2, 3, 4, 2, 11, 28, 7, 0, 0, 0]])\n",
    "src_mask = create_src_mask(src)\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.imshow(src_mask.cpu()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "타겟 문장의 마스크 생성 예는 아래와 같습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "tgt = torch.tensor([[1, 2, 3, 4, 2, 11, 28, 7, 99, 987, 1024, 0, 0]])\n",
    "enc_dec_mask, tgt_mask = create_tgt_mask(src, tgt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래 그림은 타겟 문장이 소스 문장에 Attention을 취할 때 [PAD] 토큰이 마스킹 되는 예입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f71387ef750>"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQgAAAEvCAYAAAC5X19xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAMHklEQVR4nO3cf6idB33H8fdnuWliotKKP6hJWSt03UpxVC6uWnCjUaizWP/YoIVK54T8M7WK4Nrtj/4rTERh4gi1tmBpGbHDIp1tqYoMXDH9gbaNrqVqe200FZmKQprgd3/csxGT+23ieZ7c55m8XxDuPeeee54PTfPOc849OakqJGkjfzD1AEnzZSAktQyEpJaBkNQyEJJaBkJSa2UzD3ZWttV2dm7mIaXT8kdv/PXUEybz8LeP/LSqXrPR1zY1ENvZyZ9lz2YeUjot99332NQTJrPl3Kd/2H3NhxiSWgZCUstASGoZCEktAyGpNSgQSa5M8r0kTye5caxRkuZh6UAk2QJ8BngncDFwbZKLxxomaXpDziDeDDxdVc9U1YvAXcDV48ySNAdDArELeO64y2uL635Lkr1JDiQ5cJQjAw4nabMNCUQ2uO6kt6eqqn1VtVpVq1vZNuBwkjbbkECsAecdd3k38PywOZLmZEggvgVcmOSCJGcB1wD3jDNL0hws/Y+1qupYkg8A9wFbgFur6onRlkma3KB/zVlV9wL3jrRF0sz4SkpJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGoZCEktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISS0DIallICS1DISkloGQ1DIQkloGQlLLQEhqGQhJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGotHYgk5yX5WpKDSZ5IcsOYwyRNb2XA9x4DPlpVjyR5BfBwkgeq6smRtkma2NJnEFV1qKoeWXz+S+AgsGusYZKmN8pzEEnOBy4FHhrj/iTNw5CHGAAkeTnwReDDVfWLDb6+F9gLsJ0dQw8naRMNOoNIspX1ONxRVXdvdJuq2ldVq1W1upVtQw4naZMN+SlGgM8BB6vqk+NNkjQXQ84gLgfeC1yR5LHFr78caZekGVj6OYiq+g8gI26RNDO+klJSy0BIahkISS0DIallICS1DISkloGQ1DIQkloGQlLLQEhqGQhJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGoZCEktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISS0DIallICS1DISkloGQ1DIQkloGQlJrcCCSbEnyaJIvjzFI0nyMcQZxA3BwhPuRNDODApFkN/Au4JZx5kiak6FnEJ8CPgb8ZoQtkmZm6UAkuQo4XFUPn+J2e5McSHLgKEeWPZykCQw5g7gceHeSHwB3AVck+cKJN6qqfVW1WlWrW9k24HCSNtvSgaiqm6pqd1WdD1wDfLWqrhttmaTJ+ToISa2VMe6kqr4OfH2M+5I0H55BSGoZCEktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISS0DIallICS1DISkloGQ1DIQkloGQlLLQEhqGQhJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGoZCEktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIag0KRJKzk+xP8t0kB5O8Zaxhkqa3MvD7Pw18par+KslZwI4RNkmaiaUDkeSVwNuAvwGoqheBF8eZJWkOhjzEeAPwAvD5JI8muSXJzpF2SZqBIYFYAd4EfLaqLgV+Bdx44o2S7E1yIMmBoxwZcDhJm21IINaAtap6aHF5P+vB+C1Vta+qVqtqdSvbBhxO0mZbOhBV9WPguSQXLa7aAzw5yipJszD0pxgfBO5Y/ATjGeB9wydJmotBgaiqx4DVkbZImhlfSSmpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISS0DIallICS1DISkloGQ1DIQkloGQlLLQEhqGQhJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGoZCEktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISS0DIak1KBBJPpLkiSSPJ7kzyfaxhkma3tKBSLIL+BCwWlWXAFuAa8YaJml6Qx9irAAvS7IC7ACeHz5J0lwsHYiq+hHwCeBZ4BDw86q6f6xhkqY35CHGOcDVwAXA64GdSa7b4HZ7kxxIcuAoR5ZfKmnTDXmI8Xbg+1X1QlUdBe4G3nrijapqX1WtVtXqVrYNOJykzTYkEM8ClyXZkSTAHuDgOLMkzcGQ5yAeAvYDjwDfWdzXvpF2SZqBlSHfXFU3AzePtEXSzPhKSkktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISS0DIallICS1DISkloGQ1DIQkloGQlLLQEhqGQhJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGoZCEktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISa1TBiLJrUkOJ3n8uOteleSBJE8tPp5zZmdKmsLpnEHcBlx5wnU3Ag9W1YXAg4vLkn7PnDIQVfUN4GcnXH01cPvi89uB94y8S9IMLPscxOuq6hDA4uNrx5skaS5WzvQBkuwF9gJsZ8eZPpykES17BvGTJOcCLD4e7m5YVfuqarWqVreybcnDSZrCsoG4B7h+8fn1wJfGmSNpTk7nx5x3At8ELkqyluT9wMeBdyR5CnjH4rKk3zOnfA6iqq5tvrRn5C2SZsZXUkpqGQhJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGoZCEktAyGpZSAktQyEpJaBkNQyEJJaBkJSy0BIahkISS0DIallICS1DISkloGQ1DIQkloGQlLLQEhqGQhJLQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGqdMhBJbk1yOMnjx133T0m+m+TbSf4tydlndqakKZzOGcRtwJUnXPcAcElVvRH4L+CmkXdJmoFTBqKqvgH87ITr7q+qY4uL/wnsPgPbJE1sjOcg/hb49+6LSfYmOZDkwFGOjHA4SZtlUCCS/CNwDLiju01V7auq1apa3cq2IYeTtMlWlv3GJNcDVwF7qqrGmyRpLpYKRJIrgb8H/ryqfj3uJElzcTo/5rwT+CZwUZK1JO8H/hl4BfBAkseS/MsZ3ilpAqc8g6iqaze4+nNnYIukmfGVlJJaBkJSy0BIahkISS0DIamVzXyNU5IXgB+exk1fDfz0DM8ZgzvH5c7x/C4b/7CqXrPRFzY1EKcryYGqWp16x6m4c1zuHM9YG32IIallICS15hqIfVMPOE3uHJc7xzPKxlk+ByFpHuZ6BiFpBmYXiCRXJvlekqeT3Dj1no0kOS/J15IcTPJEkhum3tRJsiXJo0m+PPWWTpKzk+xfvBHywSRvmXrTRpJ8ZPH7/XiSO5Nsn3oTtG8s/aokDyR5avHxnGXue1aBSLIF+AzwTuBi4NokF0+7akPHgI9W1Z8AlwF/N9OdADcAB6cecQqfBr5SVX8M/Ckz3JtkF/AhYLWqLgG2ANdMu+r/3MbJbyx9I/BgVV0IPLi4/DubVSCANwNPV9UzVfUicBdw9cSbTlJVh6rqkcXnv2T9f+hd0646WZLdwLuAW6be0knySuBtLN5CoKperKr/nnZVawV4WZIVYAfw/MR7gI3fWJr1Pze3Lz6/HXjPMvc9t0DsAp477vIaM/yDd7wk5wOXAg9Nu2RDnwI+Bvxm6iEv4Q3AC8DnFw+Fbkmyc+pRJ6qqHwGfAJ4FDgE/r6r7p131kl5XVYdg/S804LXL3MncApENrpvtj1mSvBz4IvDhqvrF1HuOl+Qq4HBVPTz1llNYAd4EfLaqLgV+xZKnw2fS4jH81cAFwOuBnUmum3bVmTe3QKwB5x13eTczOY07UZKtrMfhjqq6e+o9G7gceHeSH7D+UO2KJF+YdtKG1oC1qvrfM7D9rAdjbt4OfL+qXqiqo8DdwFsn3vRSfpLkXIDFx8PL3MncAvEt4MIkFyQ5i/Unge6ZeNNJkoT1x8wHq+qTU+/ZSFXdVFW7q+p81v87frWqZvc3XlX9GHguyUWLq/YAT044qfMscFmSHYvf/z3M8MnU49wDXL/4/HrgS8vcydJve38mVNWxJB8A7mP9WeJbq+qJiWdt5HLgvcB3kjy2uO4fqureCTf9f/ZB4I7FXwrPAO+beM9JquqhJPuBR1j/KdajzOQVlYs3lv4L4NVJ1oCbgY8D/7p4k+lngb9e6r59JaWkztweYkiaEQMhqWUgJLUMhKSWgZDUMhCSWgZCUstASGr9D2memfXMGijdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,5))\n",
    "plt.imshow(enc_dec_mask.cpu()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래 예는 타겟 문장에서 Self-Attention 연산이 취해질 때 타임 스텝 상 뒤에 위치하는 토큰들과 [PAD] 토큰들이 마스킹 되는 예입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f7138751c90>"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATEAAAEvCAYAAAAtufaDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAM/UlEQVR4nO3df6zddX3H8edrbWlt1YCZGmjJwISxEeLE3DiUxCVUMlQi/rE/IMOwzaT/TEVj4iD7w/8WkxmjyYxLgwiJBLJUFolxAkGNMXHE8iMIVAdBhUq1GDM1mpQS3/vjnm2X3nO5597v9/ac9/Z8JM0959vD+b5DuU8+59xvzydVhSR19XvzHkCShjBiklozYpJaM2KSWjNiklozYpJa2346T3ZGdtYu9oz6nH/4xt+O+nySFtODj574eVW99tTjpzViu9jDn2b/qM95zz2PjPp8khbTtrOf+vG0476clNSaEZPUmhGT1JoRk9SaEZPU2qCIJbkyyQ+SPJXkxrGGkqRZbTpiSbYBnwXeCVwEXJvkorEGk6RZDFmJvQV4qqqerqoXgDuBq8cZS5JmMyRie4FnV9w/Ojn2EkkOJDmc5PBJTgw4nSStNiRimXJs1cfEVtXBqlqqqqUd7BxwOklabUjEjgLnrri/D3hu2DiStDFDIvZd4IIk5yc5A7gGuHucsSRpNpv+C+BV9WKSDwD3ANuAW6rq8dEmk6QZDPoUi6r6KvDVkWaRpA3zin1JrRkxSa0ZMUmtGTFJrRkxSa2d1s/Y3wp/fs6bRn/Oe57zc/ulLlyJSWrNiElqzYhJas2ISWrNiElqzYhJas2ISWrNiElqzYhJas2ISWrNiElqzYhJas2ISWrNiElqzYhJas2ISWrNiElqzYhJas2ISWrNiElqrf1GIVvBzUekPlyJSWrNiElqzYhJas2ISWrNiElqbdMRS3Jukm8kOZLk8SQ3jDmYJM1iyCUWLwIfraqHkrwKeDDJfVX1xEizSdK6Nr0Sq6pjVfXQ5PavgSPA3rEGk6RZjPKeWJLzgEuAB8Z4Pkma1eAr9pO8EvgS8OGq+tWU3z8AHADYxe6hp5Oklxi0Ekuyg+WA3V5Vd017TFUdrKqlqlrawc4hp5OkVYb8dDLA54EjVfWp8UaSpNkNWYldBrwPuDzJI5Nf7xppLkmayabfE6uqbwMZcRZJ2jCv2JfUmhGT1JoRk9SaEZPUmhGT1JoRk9SaG4WcJm4+Im0NV2KSWjNiklozYpJaM2KSWjNiklozYpJaM2KSWjNiklozYpJaM2KSWjNiklozYpJaM2KSWjNiklozYpJaM2KSWjNiklozYpJaM2KSWjNiklpzo5DG3HxEciUmqTkjJqk1IyapNSMmqTUjJqm1wRFLsi3Jw0m+MsZAkrQRY6zEbgCOjPA8krRhgyKWZB/wbuDmccaRpI0ZuhL7NPAx4HcjzCJJG7bpiCW5CjheVQ+u87gDSQ4nOXySE5s9nSRNNWQldhnwniQ/Au4ELk/yxVMfVFUHq2qpqpZ2sHPA6SRptU1HrKpuqqp9VXUecA3w9aq6brTJJGkGXicmqbVRPsWiqr4JfHOM55KkjXAlJqk1IyapNSMmqTUjJqk1IyapNSMmqTU3CtFLuPmIunElJqk1IyapNSMmqTUjJqk1IyapNSMmqTUjJqk1IyapNSMmqTUjJqk1IyapNSMmqTUjJqk1IyapNSMmqTUjJqk1IyapNSMmqTUjJqk1P2NfW27sz+33M/u1kisxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrQ2KWJIzkxxK8v0kR5K8dazBJGkWQ68T+wzwtar6iyRnALtHmEmSZrbpiCV5NfB24K8AquoF4IVxxpKk2Qx5OfkG4HngC0keTnJzkj0jzSVJMxkSse3Am4HPVdUlwG+AG099UJIDSQ4nOXySEwNOJ0mrDYnYUeBoVT0wuX+I5ai9RFUdrKqlqlrawc4Bp5Ok1TYdsar6KfBskgsnh/YDT4wylSTNaOhPJz8I3D75yeTTwF8PH0mSZjcoYlX1CLA00iyStGFesS+pNSMmqTUjJqk1IyapNSMmqTU3ClE7Y288Am4+0pkrMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrblRiISbj3TmSkxSa0ZMUmtGTFJrRkxSa0ZMUmtGTFJrgyKW5CNJHk/yWJI7kuwaazBJmsWmI5ZkL/AhYKmqLga2AdeMNZgkzWLoy8ntwCuSbAd2A88NH0mSZrfpiFXVT4BPAs8Ax4BfVtW9Yw0mSbMY8nLyLOBq4HzgHGBPkuumPO5AksNJDp/kxOYnlaQphrycfAfww6p6vqpOAncBbzv1QVV1sKqWqmppBzsHnE6SVhsSsWeAS5PsThJgP3BknLEkaTZD3hN7ADgEPAR8b/JcB0eaS5JmMuijeKrq48DHR5pFkjbMK/YltWbEJLVmxCS1ZsQktWbEJLXmRiHSFnHzkdPDlZik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNbcKERqxM1HVnMlJqk1IyapNSMmqTUjJqk1IyapNSMmqbV1I5bkliTHkzy24thrktyX5MnJ17O2dkxJmm6WlditwJWnHLsRuL+qLgDun9yXpNNu3YhV1beAX5xy+Grgtsnt24D3jjyXJM1ks++Jvb6qjgFMvr5uvJEkaXZb/teOkhwADgDsYvdWn07S/zObXYn9LMnZAJOvx9d6YFUdrKqlqlrawc5Nnk6SpttsxO4Grp/cvh748jjjSNLGzHKJxR3Ad4ALkxxN8n7gE8AVSZ4Erpjcl6TTbt33xKrq2jV+a//Is0jShnnFvqTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNaMmKTWjJik1oyYpNbWjViSW5IcT/LYimP/mOT7SR5N8q9JztzaMSVpullWYrcCV55y7D7g4qp6I/AfwE0jzyVJM1k3YlX1LeAXpxy7t6penNz9d2DfFswmSesa4z2xvwH+ba3fTHIgyeEkh09yYoTTSdL/GhSxJH8PvAjcvtZjqupgVS1V1dIOdg45nSStsn2z/2CS64GrgP1VVeONJEmz21TEklwJ/B3wZ1X123FHkqTZzXKJxR3Ad4ALkxxN8n7gn4BXAfcleSTJP2/xnJI01borsaq6dsrhz2/BLJK0YV6xL6k1IyapNSMmqTUjJqk1IyaptZzO61STPA/8eIaH/j7w8y0eZwzOOS7nHE+HGWFjc/5BVb321IOnNWKzSnK4qpbmPcd6nHNczjmeDjPCOHP6clJSa0ZMUmuLGrGD8x5gRs45LuccT4cZYYQ5F/I9MUma1aKuxCRpJgsXsSRXJvlBkqeS3DjveaZJcm6SbyQ5kuTxJDfMe6a1JNmW5OEkX5n3LGtJcmaSQ5PNZ44keeu8Z5omyUcmf96PJbkjya55zwRrbubzmiT3JXly8vWsec44mWlLNh1aqIgl2QZ8FngncBFwbZKL5jvVVC8CH62qPwYuBf52QecEuAE4Mu8h1vEZ4GtV9UfAn7CA8ybZC3wIWKqqi4FtwDXznep/3MrqzXxuBO6vqguA+yf35+1WtmDToYWKGPAW4KmqerqqXgDuBK6e80yrVNWxqnpocvvXLH/T7Z3vVKsl2Qe8G7h53rOsJcmrgbcz+Xinqnqhqv5zvlOtaTvwiiTbgd3Ac3OeB5i+mQ/L3ze3TW7fBrz3tA41xVZtOrRoEdsLPLvi/lEWMA4rJTkPuAR4YL6TTPVp4GPA7+Y9yMt4A/A88IXJy96bk+yZ91CnqqqfAJ8EngGOAb+sqnvnO9XLen1VHYPl/+kCr5vzPLN42U2H1rJoEcuUYwv749MkrwS+BHy4qn4173lWSnIVcLyqHpz3LOvYDrwZ+FxVXQL8hsV46fMSk/eUrgbOB84B9iS5br5T/d8xy6ZDa1m0iB0Fzl1xfx8LsmQ/VZIdLAfs9qq6a97zTHEZ8J4kP2L5ZfnlSb4435GmOgocrar/XskeYjlqi+YdwA+r6vmqOgncBbxtzjO9nJ8lORtg8vX4nOdZ04pNh/5yM5sOLVrEvgtckOT8JGew/Mbp3XOeaZUkYfk9nCNV9al5zzNNVd1UVfuq6jyW/z1+vaoWbuVQVT8Fnk1y4eTQfuCJOY60lmeAS5Psnvz572cBfwCxwt3A9ZPb1wNfnuMsa1qx6dB7Nrvp0EJFbPIG3weAe1j+D+Rfqurx+U411WXA+1he3Twy+fWueQ/V2AeB25M8CrwJ+Ic5z7PKZKV4CHgI+B7L3zsLcVX8Gpv5fAK4IsmTwBWT+3O1VZsOecW+pNYWaiUmSRtlxCS1ZsQktWbEJLVmxCS1ZsQktWbEJLVmxCS19l8GAeY5Ui708gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,5))\n",
    "plt.imshow(tgt_mask.cpu()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5-2. Position-wise Feed-Forward 네트워크 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/positionwise.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## 5-2. Position-wise Feed-Forward 네트워크 구현class PositionwiseFeedForward(nn.Module):\n",
    "    '''포지션 와이즈 피드 포워드 레이어'''\n",
    "    def __init__(self, parmas):\n",
    "        super(PositionwiseFeedForward, self).__init__()\n",
    "        self.fc1 = nn.Linear(params['hidden_dim'], params['ffn_dim'])\n",
    "        self.fc2 = nn.Linear(params['ffn_dim'], params['hidden_dim'])\n",
    "        self.dropout = nn.Dropout(params['dropout'])\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \" x = [배치 사이즈, 문장 길이, 은닉 차원] \"\n",
    "\n",
    "        x = self.dropout(F.relu(self.fc1(x)))\n",
    "        x = self.fc2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5-3. Positional Encoding 레이어 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/pos.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PositionalEncoding(nn.Module):\n",
    "    def __init__(self, params):\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        sinusoid = np.array([pos / np.power(10000, 2 * i / params['hidden_dim'])\n",
    "                            for pos in range(params['max_len']) for i in range(params['hidden_dim'])])\n",
    "        # sinusoid = [문장 최대 길이 * 은닉 차원]\n",
    "\n",
    "        sinusoid = sinusoid.reshape(params['max_len'], -1)\n",
    "        # sinusoid = [문장 최대 길이, 은닉 차원]\n",
    "\n",
    "        sinusoid[:, 0::2] = np.sin(sinusoid[:, 0::2])\n",
    "        sinusoid[:, 1::2] = np.cos(sinusoid[:, 1::2])\n",
    "        sinusoid = torch.FloatTensor(sinusoid).to(device)\n",
    "\n",
    "        self.embedding = nn.Embedding.from_pretrained(sinusoid, freeze=True)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        \" x = [배치 사이즈, 문장 길이] \"\n",
    "        \n",
    "        pos = torch.arange(x.size(-1), dtype=torch.long).to(device)\n",
    "        # pos = [배치 사이즈, 문장 길이]\n",
    "\n",
    "        embed = self.embedding(pos)\n",
    "        # embed = [배치 사이즈, 문장 길이, 은닉 차원]\n",
    "        return embed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5-4. Transformer 인코더 부 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/encoder.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderLayer(nn.Module):\n",
    "    '''인코더 레이어'''\n",
    "    def __init__(self, params):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "        self.self_attn = MultiHeadAttention(params)\n",
    "        self.layer_norm1 = nn.LayerNorm(params['hidden_dim'])\n",
    "        self.feed_forward = PositionwiseFeedForward(params)\n",
    "        self.layer_norm2 = nn.LayerNorm(params['hidden_dim'])\n",
    "        self.dropout = nn.Dropout(params['dropout'])\n",
    "        \n",
    "    def forward(self, x, src_mask):\n",
    "        \" x = [배치 사이즈, 문장 길이, 은닉 차원] \"\n",
    "        \n",
    "        residual = x\n",
    "        x, _ = self.self_attn(x, x, x, src_mask)\n",
    "        x = self.dropout(x)\n",
    "        x = residual + x\n",
    "        x = self.layer_norm1(x)\n",
    "        \n",
    "        residual = x\n",
    "        x = self.feed_forward(x)\n",
    "        x = self.dropout(x)\n",
    "        x = residual + x\n",
    "        x = self.layer_norm2(x)\n",
    "        \n",
    "        return x\n",
    "\n",
    "\n",
    "class Encoder(nn.Module):\n",
    "    '''트랜스포머 인코더'''\n",
    "    def __init__(self, params):\n",
    "        super(Encoder, self).__init__()\n",
    "        self.tok_embedding = nn.Embedding(params['vocab_size'], params['hidden_dim'], padding_idx=pad_idx)\n",
    "        self.pos_embedding = PositionalEncoding(params)\n",
    "        self.layers = nn.ModuleList([EncoderLayer(params) for _ in range(params['num_layers'])])\n",
    "        \n",
    "    def forward(self, src):\n",
    "        \" src = [배치 사이즈, 소스 문장 길이] \"\n",
    "\n",
    "        src_mask = create_src_mask(src)\n",
    "        src = self.tok_embedding(src) + self.pos_embedding(src)\n",
    "        \n",
    "        for layer in self.layers:\n",
    "            src = layer(src, src_mask)\n",
    "            \n",
    "        # src = [배치 사이즈, 소스 문장 길이, 은닉 차원]\n",
    "        return src"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5-5. Transformer 인코더 부 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/decoder.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DecoderLayer(nn.Module):\n",
    "    '''디코더 레이어'''\n",
    "    def __init__(self, params):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "        self.self_attn = MultiHeadAttention(params)\n",
    "        self.layer_norm1 = nn.LayerNorm(params['hidden_dim'])\n",
    "\n",
    "        self.enc_dec_attn = MultiHeadAttention(params)\n",
    "        self.layer_norm2 = nn.LayerNorm(params['hidden_dim'])\n",
    "        \n",
    "        self.feed_forward = PositionwiseFeedForward(params)\n",
    "        self.layer_norm3 = nn.LayerNorm(params['hidden_dim'])\n",
    "        \n",
    "        self.dropout = nn.Dropout(params['dropout'])\n",
    "        \n",
    "    def forward(self, x, tgt_mask, enc_output, src_mask):\n",
    "        \" x = [배치 사이즈, 문장 길이, 은닉 차원] \"\n",
    "        \n",
    "        residual = x\n",
    "        x, _ = self.self_attn(x, x, x, tgt_mask)\n",
    "        x = self.dropout(x)\n",
    "        x = residual + x\n",
    "        x = self.layer_norm1(x)\n",
    "        \n",
    "        residual = x\n",
    "        x, attn_map = self.enc_dec_attn(x, enc_output, enc_output, src_mask)\n",
    "        x = self.dropout(x)\n",
    "        x = residual + x\n",
    "        x = self.layer_norm2(x)\n",
    "        \n",
    "        residual = x\n",
    "        x = self.feed_forward(x)\n",
    "        x = self.dropout(x)\n",
    "        x = residual + x\n",
    "        x = self.layer_norm3(x)\n",
    "        \n",
    "        return x, attn_map\n",
    "\n",
    "\n",
    "class Decoder(nn.Module):\n",
    "    '''트랜스포머 디코더'''\n",
    "    def __init__(self, params):\n",
    "        super(Decoder, self).__init__()\n",
    "        self.tok_embedding = nn.Embedding(params['vocab_size'], params['hidden_dim'], padding_idx=pad_idx)\n",
    "        self.pos_embedding = PositionalEncoding(params)\n",
    "        self.layers = nn.ModuleList([DecoderLayer(params) for _ in range(params['num_layers'])])\n",
    "        \n",
    "    def forward(self, tgt, src, enc_out):\n",
    "        \" tgt = [배치 사이즈, 타겟 문장 길이] \"\n",
    "\n",
    "        src_mask, tgt_mask = create_tgt_mask(src, tgt)\n",
    "        tgt = self.tok_embedding(tgt) + self.pos_embedding(tgt)\n",
    "        \n",
    "        for layer in self.layers:\n",
    "            tgt, attn_map = layer(tgt, tgt_mask, enc_out, src_mask)\n",
    "            \n",
    "        tgt = torch.matmul(tgt, self.tok_embedding.weight.transpose(0, 1))\n",
    "        # tgt = [배치 사이즈, 타겟 문장 길이, 은닉 차원]\n",
    "\n",
    "        return tgt, attn_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5-6. Transformer 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 앞서 정의한 레이어들을 토대로 **Transformer** 모델을 빌드해줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Transformer(nn.Module):\n",
    "    '''트랜스포머 네트워크'''\n",
    "    def __init__(self, params):\n",
    "        super(Transformer, self).__init__()\n",
    "        self.encoder = Encoder(params)\n",
    "        self.decoder = Decoder(params)\n",
    "    \n",
    "    def forward(self, src, tgt):\n",
    "        \" src = [배치 사이즈, 소스 문장 길이] \"\n",
    "        \" tgt = [배치 사이즈, 타겟 문장 길이] \"\n",
    "        \n",
    "        enc_out = self.encoder(src)\n",
    "        dec_out, attn = self.decoder(tgt, src, enc_out)\n",
    "        return dec_out, attn\n",
    "    \n",
    "    def count_params(self):\n",
    "        return sum(p.numel() for p in self.parameters() if p.requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transformer 에서는 Adam Optimizer에 일부 스케줄 옵션을 적용해 사용하고 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5-7. Transformer Optimizer 구현"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/optim.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ScheduledOptim:\n",
    "    '''스케줄 옵티마이저'''\n",
    "    def __init__(self, optimizer, warmup_steps):\n",
    "        self.init_lr = np.power(params['hidden_dim'], -0.5)\n",
    "        self.optimizer = optimizer\n",
    "        self.step_num = 0\n",
    "        self.warmup_steps = warmup_steps\n",
    "    \n",
    "    def step(self):\n",
    "        self.step_num += 1\n",
    "        lr = self.init_lr * self.get_scale()\n",
    "        \n",
    "        for p in self.optimizer.param_groups:\n",
    "            p['lr'] = lr\n",
    "            \n",
    "        self.optimizer.step()\n",
    "    \n",
    "    def zero_grad(self):\n",
    "        self.optimizer.zero_grad()\n",
    "    \n",
    "    def get_scale(self):\n",
    "        return np.min([\n",
    "            np.power(self.step_num, -0.5),\n",
    "            self.step_num * np.power(self.warmup_steps, -1.5)\n",
    "        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 모델 학습"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 64,618,496 trainable parameters\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-185-9deae2ff857e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     32\u001b[0m         \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 34\u001b[0;31m         \u001b[0mlogits\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     35\u001b[0m         \u001b[0;31m# logits = [배치 사이즈, 타겟 문장 길이, 은닉 차원]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     36\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    530\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    531\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 532\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    533\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    534\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-129-dbffa90adff9>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, src, tgt)\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m         \u001b[0menc_out\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m         \u001b[0mdec_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecoder\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtgt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0menc_out\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mdec_out\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    530\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    531\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 532\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    533\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    534\u001b[0m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-128-5174c9f860f2>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, tgt, src, enc_out)\u001b[0m\n\u001b[1;32m     49\u001b[0m         \u001b[0;34m\" tgt = [배치 사이즈, 타겟 문장 길이] \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 51\u001b[0;31m         \u001b[0msrc_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_tgt_mask\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m         \u001b[0mtgt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtok_embedding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtgt\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpos_embedding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtgt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-157-01dc49227d8c>\u001b[0m in \u001b[0;36mcreate_tgt_mask\u001b[0;34m(src, tgt)\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_len\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtgt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m     \u001b[0msubsequent_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_subsequent_mask\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtgt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m     \u001b[0menc_dec_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mpad_idx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-136-878906d90e6b>\u001b[0m in \u001b[0;36mcreate_subsequent_mask\u001b[0;34m(tgt)\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0;31m# subsequent_mask = [타겟 문장 길이, 타겟 문장 길이]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m     \u001b[0msubsequent_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubsequent_mask\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munsqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrepeat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m     \u001b[0;31m# subsquent_mask = [배치 사이즈, 타겟 문장 길이, 타겟 문장 길이]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# 모델 정의\n",
    "\n",
    "model = Transformer(params)\n",
    "model.to(device)\n",
    "print(f'The model has {model.count_params():,} trainable parameters')\n",
    "\n",
    "\n",
    "# 로스 함수 정의\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index=pad_idx)\n",
    "criterion.to(device)\n",
    "\n",
    "\n",
    "# 옵티마이저 정의\n",
    "\n",
    "optimizer = ScheduledOptim(\n",
    "    optim.Adam(model.parameters(), betas=[0.9, 0.98], eps=1e-9),\n",
    "    warmup_steps=4000\n",
    ")\n",
    "\n",
    "\n",
    "# 훈련 로직\n",
    "\n",
    "for epoch in range(params['num_epoch']):\n",
    "    model.train()\n",
    "    epoch_loss = 0\n",
    "\n",
    "    for src, tgt in zip(src_iter, tgt_iter):\n",
    "        \" src = [배치 사이즈, 소스 문장 길이] \"\n",
    "        \" tgt = [배치 사이즈, 타겟 문장 길이] \"\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        logits, _ = model(src, tgt[:, :-1])\n",
    "        # logits = [배치 사이즈, 타겟 문장 길이, 은닉 차원]\n",
    "        \n",
    "        logits = logits.contiguous().view(-1, logits.size(-1))\n",
    "        # logits = [(배치 사이즈 * 타겟 문장 길이) - 1, 은닉 차원]\n",
    "        golds = tgt[:, 1:].contiguous().view(-1)\n",
    "        # golds = [(배치 사이즈 * 타겟 문장 길이) - 1]\n",
    "\n",
    "        loss = criterion(logits, golds)\n",
    "        epoch_loss += loss.item()\n",
    "        \n",
    "        loss.backward()\n",
    "#         torch.nn.utils.clip_grad_norm_(model.parameters(), self.params.clip)\n",
    "        optimizer.step()\n",
    "\n",
    "    train_loss = epoch_loss / len(self.train_iter)\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 학습 결과 확인"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 학습된 모델이 어느 정도 성능을 보이는지 확인해 볼 차례입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict():\n",
    "    model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 참고자료\n",
    "- [The Annotated Transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html)\n",
    "- [jadore801120/attention-is-all-you-need-pytorch](https://github.com/jadore801120/attention-is-all-you-need-pytorch)\n",
    "- [tunz/transformer-pytorch](https://github.com/tunz/transformer-pytorch)\n",
    "- [IgorSusmelj/pytorch-styleguide](https://github.com/IgorSusmelj/pytorch-styleguide)\n",
    "\n",
    "\n",
    "### TODO: 더 추가할 수 있는 것들 !\n",
    "- **Beam Search** 디코딩 추가\n",
    "- **Label Smoothing** 기법 추가"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}