exercise-deep-reinforcement-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.007926,
     "end_time": "2020-10-01T00:25:51.178752",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.170826",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Intro to Game AI and Reinforcement Learning](https://www.kaggle.com/learn/intro-to-game-ai-and-reinforcement-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/deep-reinforcement-learning).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.006329,
     "end_time": "2020-10-01T00:25:51.191981",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.185652",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction\n",
    "\n",
    "In the tutorial, you learned a bit about reinforcement learning and used the `stable-baselines` package to train an agent to beat a random opponent.  In this exercise, you will check your understanding and tinker with the code to deepen your intuition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-10-01T00:25:51.210887Z",
     "iopub.status.busy": "2020-10-01T00:25:51.210147Z",
     "iopub.status.idle": "2020-10-01T00:25:51.261278Z",
     "shell.execute_reply": "2020-10-01T00:25:51.260577Z"
    },
    "papermill": {
     "duration": 0.062774,
     "end_time": "2020-10-01T00:25:51.261396",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.198622",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.game_ai.ex4 import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.006843,
     "end_time": "2020-10-01T00:25:51.275366",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.268523",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 1) Set the architecture\n",
    "\n",
    "In the tutorial, you learned one way to design a neural network that can select moves in Connect Four.  The neural network had an output layer with seven nodes: one for each column in the game board.\n",
    "\n",
    "Say now you wanted to create a neural network that can play chess.  How many nodes should you put in the output layer?\n",
    "\n",
    "- Option A: 2 nodes (number of game players)\n",
    "- Option B: 16 nodes (number of game pieces that each player starts with)\n",
    "- Option C: 4672 nodes (number of possible moves)\n",
    "- Option D: 64 nodes (number of squares on the game board)\n",
    "\n",
    "Use your answer to set the value of the `best_option` variable below.  Your answer should be one of `'A'`, `'B'`, `'C'`, or `'D'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-10-01T00:25:51.295981Z",
     "iopub.status.busy": "2020-10-01T00:25:51.295149Z",
     "iopub.status.idle": "2020-10-01T00:25:51.301781Z",
     "shell.execute_reply": "2020-10-01T00:25:51.301308Z"
    },
    "papermill": {
     "duration": 0.019587,
     "end_time": "2020-10-01T00:25:51.301882",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.282295",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 4, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1_PickBestOption\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#ccaa33\">Check:</span> When you've updated the starter code, `check()` will tell you whether your code is correct. You need to update the code that creates variable `best_option`"
      ],
      "text/plain": [
       "Check: When you've updated the starter code, `check()` will tell you whether your code is correct. You need to update the code that creates variable `best_option`"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the blank\n",
    "best_option = ____\n",
    "\n",
    "# Check your answer\n",
    "q_1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-10-01T00:25:51.320326Z",
     "iopub.status.busy": "2020-10-01T00:25:51.319774Z",
     "iopub.status.idle": "2020-10-01T00:25:51.322600Z",
     "shell.execute_reply": "2020-10-01T00:25:51.323108Z"
    },
    "papermill": {
     "duration": 0.013489,
     "end_time": "2020-10-01T00:25:51.323248",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.309759",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you solution code\n",
    "#q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.007182,
     "end_time": "2020-10-01T00:25:51.339687",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.332505",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 2) Decide reward\n",
    "\n",
    "In the tutorial, you learned how to give your agent a reward that encourages it to win games of Connect Four.  Consider now training an agent to win at the game [Minesweeper](https://bit.ly/2T5xEY8).  The goal of the game is to clear the board without detonating any bombs.\n",
    "\n",
    "To play this game in Google Search, click on the **[Play]** button at [this link](https://www.google.com/search?q=minesweeper).  \n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/WzoEfKY.png\" width=50%><br/>\n",
    "</center>\n",
    "\n",
    "With each move, one of the following is true:\n",
    "- The agent selected an invalid move (in other words, it tried to uncover a square that was uncovered as part of a previous move).  Let's assume this ends the game, and the agent loses.\n",
    "- The agent clears a square that did not contain a hidden mine.  The agent wins the game, because all squares without mines are revealed.\n",
    "- The agent clears a square that did not contain a hidden mine, but has not yet won or lost the game.\n",
    "- The agent detonates a mine and loses the game.\n",
    "\n",
    "How might you specify the reward for each of these four cases, so that by maximizing the cumulative reward, the agent will try to win the game?\n",
    "\n",
    "After you have decided on your answer, run the code cell below to get credit for completing this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-10-01T00:25:51.357777Z",
     "iopub.status.busy": "2020-10-01T00:25:51.357086Z",
     "iopub.status.idle": "2020-10-01T00:25:51.362998Z",
     "shell.execute_reply": "2020-10-01T00:25:51.363540Z"
    },
    "papermill": {
     "duration": 0.01674,
     "end_time": "2020-10-01T00:25:51.363700",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.346960",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 4, \"questionId\": \"2_DecideReward\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> Here's a possible solution - after each move, we give the agent a reward that tells it how well it did:\n",
       "- If agent wins the game in that move, it gets a reward of `+1`.\n",
       "- Else if the agent selects an invalid move, it gets a reward of `-10`.\n",
       "- Else if it detonates a mine, it gets a reward of `-1`.\n",
       "- Else if the agent clears a square with no hidden mine, it gets a reward of `+1/100`.\n",
       "\n",
       "To check the validity of your answer, note that the reward for selecting an invalid move and for detonating a mine should both be negative.  The reward for winning the game should be positive.  And, the reward for clearing a square with no hidden mine should be either zero or slightly positive."
      ],
      "text/plain": [
       "Solution: Here's a possible solution - after each move, we give the agent a reward that tells it how well it did:\n",
       "- If agent wins the game in that move, it gets a reward of `+1`.\n",
       "- Else if the agent selects an invalid move, it gets a reward of `-10`.\n",
       "- Else if it detonates a mine, it gets a reward of `-1`.\n",
       "- Else if the agent clears a square with no hidden mine, it gets a reward of `+1/100`.\n",
       "\n",
       "To check the validity of your answer, note that the reward for selecting an invalid move and for detonating a mine should both be negative.  The reward for winning the game should be positive.  And, the reward for clearing a square with no hidden mine should be either zero or slightly positive."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.008155,
     "end_time": "2020-10-01T00:25:51.380579",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.372424",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### 3) (Optional) Amend the code\n",
    "\n",
    "In this next part of the exercise, you will amend the code from the tutorial to experiment with creating your own agents!  There are a lot of hyperparameters involved with specifying a reinforcement learning agent, and you'll have a chance to amend them, to see how performance is affected.\n",
    "\n",
    "First, we'll need to make sure that your Kaggle Notebook is set up to run the code.  Begin by looking at the \"Settings\" menu to the right of your notebook.  Your menu will look like one of the following:\n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/kR1az0y.png\" width=100%><br/>\n",
    "</center>\n",
    "\n",
    "If your \"Internet\" setting appears as a \"Requires phone verification\" link, click on this link.  This will bring you to a new window; then, follow the instructions to verify your account.  After following this step, your \"Internet\" setting will appear \"Off\", as in the example to the right.\n",
    "\n",
    "Once your \"Internet\" setting appears as \"Off\", click to turn it on.  You'll see a pop-up window that you'll need to \"Accept\" in order to complete the process and have the setting switched to \"On\".  Once the Internet is turned \"On\", you're ready to proceed!\n",
    "\n",
    "<center>\n",
    "<img src=\"https://i.imgur.com/gOVh6Aa.png\" width=100%><br/>\n",
    "</center>\n",
    "\n",
    "Begin by running the code cell below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-10-01T00:25:51.400919Z",
     "iopub.status.busy": "2020-10-01T00:25:51.400212Z",
     "iopub.status.idle": "2020-10-01T00:42:33.327153Z",
     "shell.execute_reply": "2020-10-01T00:42:33.327712Z"
    },
    "papermill": {
     "duration": 1001.939006,
     "end_time": "2020-10-01T00:42:33.327873",
     "exception": false,
     "start_time": "2020-10-01T00:25:51.388867",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdc77b3dad0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/tensorflow/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdc77b98f10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/tensorflow/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdc77b98b90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/tensorflow/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdc77b98d50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/tensorflow/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdc77b8a1d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/tensorflow/\u001b[0m\r\n",
      "\u001b[31mERROR: Could not find a version that satisfies the requirement tensorflow==1.15.0 (from versions: none)\u001b[0m\r\n",
      "\u001b[31mERROR: No matching distribution found for tensorflow==1.15.0\u001b[0m\r\n",
      "Failed: football: No module named 'gfootball'\n",
      "Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:2 http://security.ubuntu.com/ubuntu bionic-security InRelease\r\n",
      "  Temporary failure resolving 'security.ubuntu.com'\r\n",
      "Err:3 http://packages.cloud.google.com/apt gcsfuse-bionic InRelease\r\n",
      "  Temporary failure resolving 'packages.cloud.google.com'\r\n",
      "Err:4 http://packages.cloud.google.com/apt cloud-sdk-bionic InRelease\r\n",
      "  Temporary failure resolving 'packages.cloud.google.com'\r\n",
      "Err:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:6 http://packages.cloud.google.com/apt cloud-sdk InRelease\r\n",
      "  Temporary failure resolving 'packages.cloud.google.com'\r\n",
      "Err:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "\r\n",
      "W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease  Temporary failure resolving 'security.ubuntu.com'\r\n",
      "W: Failed to fetch http://packages.cloud.google.com/apt/dists/gcsfuse-bionic/InRelease  Temporary failure resolving 'packages.cloud.google.com'\r\n",
      "W: Failed to fetch http://packages.cloud.google.com/apt/dists/cloud-sdk-bionic/InRelease  Temporary failure resolving 'packages.cloud.google.com'\r\n",
      "W: Failed to fetch http://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease  Temporary failure resolving 'packages.cloud.google.com'\r\n",
      "W: Some index files failed to download. They have been ignored, or old ones used instead.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "zlib1g-dev is already the newest version (1:1.2.11.dfsg-0ubuntu2).\r\n",
      "cmake is already the newest version (3.10.2-1ubuntu2.18.04.1).\r\n",
      "python3-dev is already the newest version (3.6.7-1~18.04).\r\n",
      "python3-dev set to manually installed.\r\n",
      "The following additional packages will be installed:\r\n",
      "  autotools-dev ibverbs-providers libfabric1 libhwloc-dev libhwloc-plugins\r\n",
      "  libhwloc5 libibverbs-dev libibverbs1 libltdl-dev libnl-3-200\r\n",
      "  libnl-route-3-200 libnuma-dev libopenmpi2 libpsm-infinipath1 librdmacm1\r\n",
      "  libtool ocl-icd-libopencl1 openmpi-bin openmpi-common\r\n",
      "Suggested packages:\r\n",
      "  libhwloc-contrib-plugins libtool-doc openmpi-doc autoconf automaken gfortran\r\n",
      "  | fortran95-compiler gcj-jdk opencl-icd gfortran\r\n",
      "The following NEW packages will be installed:\r\n",
      "  autotools-dev ibverbs-providers libfabric1 libhwloc-dev libhwloc-plugins\r\n",
      "  libhwloc5 libibverbs-dev libibverbs1 libltdl-dev libnl-3-200\r\n",
      "  libnl-route-3-200 libnuma-dev libopenmpi-dev libopenmpi2 libpsm-infinipath1\r\n",
      "  librdmacm1 libtool ocl-icd-libopencl1 openmpi-bin openmpi-common\r\n",
      "0 upgraded, 20 newly installed, 0 to remove and 89 not upgraded.\r\n",
      "Need to get 4991 kB of archives.\r\n",
      "After this operation, 20.7 MB of additional disk space will be used.\r\n",
      "Err:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 autotools-dev all 20180224.1\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libnl-3-200 amd64 3.2.29-0ubuntu3\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 libnl-route-3-200 amd64 3.2.29-0ubuntu3\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libibverbs1 amd64 17.1-1ubuntu0.2\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ibverbs-providers amd64 17.1-1ubuntu0.2\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libpsm-infinipath1 amd64 3.3+20.604758e7-5\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librdmacm1 amd64 17.1-1ubuntu0.2\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libfabric1 amd64 1.5.3-1\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libltdl-dev amd64 2.4.6-2\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libtool all 2.4.6-2\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libhwloc5 amd64 1.11.9-1\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 ocl-icd-libopencl1 amd64 2.2.11-1ubuntu1\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:13 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libhwloc-plugins amd64 1.11.9-1\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libopenmpi2 amd64 2.1.1-8\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 openmpi-common all 2.1.1-8\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 openmpi-bin amd64 2.1.1-8\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnuma-dev amd64 2.0.11-2.1ubuntu0.1\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libhwloc-dev amd64 1.11.9-1\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libibverbs-dev amd64 17.1-1ubuntu0.2\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "Err:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libopenmpi-dev amd64 2.1.1-8\r\n",
      "  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/a/autotools-dev/autotools-dev_20180224.1_all.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libn/libnl3/libnl-3-200_3.2.29-0ubuntu3_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libn/libnl3/libnl-route-3-200_3.2.29-0ubuntu3_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/r/rdma-core/libibverbs1_17.1-1ubuntu0.2_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/r/rdma-core/ibverbs-providers_17.1-1ubuntu0.2_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/i/infinipath-psm/libpsm-infinipath1_3.3+20.604758e7-5_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/r/rdma-core/librdmacm1_17.1-1ubuntu0.2_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/libf/libfabric/libfabric1_1.5.3-1_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libt/libtool/libltdl-dev_2.4.6-2_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libt/libtool/libtool_2.4.6-2_all.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/h/hwloc/libhwloc5_1.11.9-1_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/o/ocl-icd/ocl-icd-libopencl1_2.2.11-1ubuntu1_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/h/hwloc/libhwloc-plugins_1.11.9-1_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/o/openmpi/libopenmpi2_2.1.1-8_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/o/openmpi/openmpi-common_2.1.1-8_all.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/o/openmpi/openmpi-bin_2.1.1-8_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/n/numactl/libnuma-dev_2.0.11-2.1ubuntu0.1_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/h/hwloc/libhwloc-dev_1.11.9-1_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/r/rdma-core/libibverbs-dev_17.1-1ubuntu0.2_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/o/openmpi/libopenmpi-dev_2.1.1-8_amd64.deb  Temporary failure resolving 'archive.ubuntu.com'\r\n",
      "E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfaf00ab10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/stable-baselines/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfaf113990>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/stable-baselines/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfaf1130d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/stable-baselines/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfaf1136d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/stable-baselines/\u001b[0m\r\n",
      "\u001b[33mWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fdfaf113410>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/stable-baselines/\u001b[0m\r\n",
      "\u001b[31mERROR: Could not find a version that satisfies the requirement stable-baselines[mpi]==2.9.0 (from versions: none)\u001b[0m\r\n",
      "\u001b[31mERROR: No matching distribution found for stable-baselines[mpi]==2.9.0\u001b[0m\r\n"
     ]
    },
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'stable_baselines'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-811825f76958>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msystem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'pip install \"stable-baselines[mpi]==2.9.0\"'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mstable_baselines\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbench\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mMonitor\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     19\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mstable_baselines\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcommon\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvec_env\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDummyVecEnv\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mstable_baselines\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mPPO1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mA2C\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mACER\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mACKTR\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTRPO\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'stable_baselines'"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import random\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "!pip install 'tensorflow==1.15.0'\n",
    "\n",
    "import tensorflow as tf\n",
    "from kaggle_environments import make, evaluate\n",
    "from gym import spaces\n",
    "\n",
    "!apt-get update\n",
    "!apt-get install -y cmake libopenmpi-dev python3-dev zlib1g-dev\n",
    "!pip install \"stable-baselines[mpi]==2.9.0\"\n",
    "\n",
    "from stable_baselines.bench import Monitor \n",
    "from stable_baselines.common.vec_env import DummyVecEnv\n",
    "from stable_baselines import PPO1, A2C, ACER, ACKTR, TRPO\n",
    "from stable_baselines.a2c.utils import conv, linear, conv_to_fc\n",
    "from stable_baselines.common.policies import CnnPolicy\n",
    "\n",
    "class ConnectFourGym:\n",
    "    def __init__(self, agent2=\"random\"):\n",
    "        ks_env = make(\"connectx\", debug=True)\n",
    "        self.env = ks_env.train([None, agent2])\n",
    "        self.rows = ks_env.configuration.rows\n",
    "        self.columns = ks_env.configuration.columns\n",
    "        # Learn about spaces here: http://gym.openai.com/docs/#spaces\n",
    "        self.action_space = spaces.Discrete(self.columns)\n",
    "        self.observation_space = spaces.Box(low=0, high=2, \n",
    "                                            shape=(self.rows,self.columns,1), dtype=np.int)\n",
    "        # Tuple corresponding to the min and max possible rewards\n",
    "        self.reward_range = (-10, 1)\n",
    "        # StableBaselines throws error if these are not defined\n",
    "        self.spec = None\n",
    "        self.metadata = None\n",
    "    def reset(self):\n",
    "        self.obs = self.env.reset()\n",
    "        return np.array(self.obs['board']).reshape(self.rows,self.columns,1)\n",
    "    def change_reward(self, old_reward, done):\n",
    "        if old_reward == 1: # The agent won the game\n",
    "            return 1\n",
    "        elif done: # The opponent won the game\n",
    "            return -1\n",
    "        else: # Reward 1/42\n",
    "            return 1/(self.rows*self.columns)\n",
    "    def step(self, action):\n",
    "        # Check if agent's move is valid\n",
    "        is_valid = (self.obs['board'][int(action)] == 0)\n",
    "        if is_valid: # Play the move\n",
    "            self.obs, old_reward, done, _ = self.env.step(int(action))\n",
    "            reward = self.change_reward(old_reward, done)\n",
    "        else: # End the game and penalize agent\n",
    "            reward, done, _ = -10, True, {}\n",
    "        return np.array(self.obs['board']).reshape(self.rows,self.columns,1), reward, done, _\n",
    "    \n",
    "# Create ConnectFour environment\n",
    "env = ConnectFourGym(agent2=\"random\")\n",
    "\n",
    "# Create directory for logging training information\n",
    "log_dir = \"log/\"\n",
    "os.makedirs(log_dir, exist_ok=True)\n",
    "\n",
    "# Logging progress\n",
    "monitor_env = Monitor(env, log_dir, allow_early_resets=True)\n",
    "\n",
    "# Create a vectorized environment\n",
    "vec_env = DummyVecEnv([lambda: monitor_env])\n",
    "\n",
    "# Neural network for predicting action values\n",
    "def modified_cnn(scaled_images, **kwargs):\n",
    "    activ = tf.nn.relu\n",
    "    layer_1 = activ(conv(scaled_images, 'c1', n_filters=32, filter_size=3, stride=1, \n",
    "                         init_scale=np.sqrt(2), **kwargs))\n",
    "    layer_2 = activ(conv(layer_1, 'c2', n_filters=64, filter_size=3, stride=1, \n",
    "                         init_scale=np.sqrt(2), **kwargs))\n",
    "    layer_2 = conv_to_fc(layer_2)\n",
    "    return activ(linear(layer_2, 'fc1', n_hidden=512, init_scale=np.sqrt(2)))  \n",
    "\n",
    "class CustomCnnPolicy(CnnPolicy):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(CustomCnnPolicy, self).__init__(*args, **kwargs, cnn_extractor=modified_cnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.338451,
     "end_time": "2020-10-01T00:42:34.002958",
     "exception": false,
     "start_time": "2020-10-01T00:42:33.664507",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Next, run the code cell below to train an agent with PPO and view how the rewards evolved during training.  This code is identical to the code from the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-10-01T00:42:34.682278Z",
     "iopub.status.busy": "2020-10-01T00:42:34.681685Z",
     "iopub.status.idle": "2020-10-01T00:42:34.684770Z",
     "shell.execute_reply": "2020-10-01T00:42:34.685236Z"
    },
    "papermill": {
     "duration": 0.354775,
     "end_time": "2020-10-01T00:42:34.685450",
     "exception": false,
     "start_time": "2020-10-01T00:42:34.330675",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'PPO1' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-5fd9c9346f5a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Initialize agent\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mPPO1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mCustomCnnPolicy\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvec_env\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# Train agent\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'PPO1' is not defined"
     ]
    }
   ],
   "source": [
    "# Initialize agent\n",
    "model = PPO1(CustomCnnPolicy, vec_env, verbose=0)\n",
    "\n",
    "# Train agent\n",
    "model.learn(total_timesteps=100000)\n",
    "\n",
    "# Plot cumulative reward\n",
    "with open(os.path.join(log_dir, \"monitor.csv\"), 'rt') as fh:    \n",
    "    firstline = fh.readline()\n",
    "    assert firstline[0] == '#'\n",
    "    df = pd.read_csv(fh, index_col=None)['r']\n",
    "df.rolling(window=1000).mean().plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.328226,
     "end_time": "2020-10-01T00:42:35.352465",
     "exception": false,
     "start_time": "2020-10-01T00:42:35.024239",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "If your agent trained well, the plot (which shows average cumulative rewards) should increase over time.\n",
    "\n",
    "Once you have verified that the code runs, try making amendments to see if you can get increased performance.  You might like to:\n",
    "- change `PPO1` to `A2C` (or `ACER` or `ACKTR` or `TRPO`) when defining the model in this line of code: `model = PPO1(CustomCnnPolicy, vec_env, verbose=0)`.  This will let you see how performance can be affected by changing the algorithm from Proximal Policy Optimization [PPO] to one of:\n",
    "  - Advantage Actor-Critic (A2C),\n",
    "  - or Actor-Critic with Experience Replay (ACER),\n",
    "  - Actor Critic using Kronecker-factored Trust Region (ACKTR), or \n",
    "  - Trust Region Policy Optimization (TRPO).\n",
    "- modify the `change_reward()` method in the `ConnectFourGym` class to change the rewards that the agent receives in different conditions.  You may also need to modify `self.reward_range` in the `__init__` method (this tuple should always correspond to the minimum and maximum reward that the agent can receive).\n",
    "- change `agent2` to a different agent when creating the ConnectFour environment with `env = ConnectFourGym(agent2=\"random\")`.  For instance, you might like to use the `\"negamax\"` agent, or a different, custom agent.  Note that the smarter you make the opponent, the harder it will be for your agent to train!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.329712,
     "end_time": "2020-10-01T00:42:36.009641",
     "exception": false,
     "start_time": "2020-10-01T00:42:35.679929",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Congratulations!\n",
    "\n",
    "You have completed the course, and it's time to put your new skills to work!  \n",
    "\n",
    "The next step is to apply what you've learned to a **[more complex game: Halite](https://www.kaggle.com/c/halite)**.  For a step-by-step tutorial in how to make your first submission to this competition, **[check out the bonus lesson](https://www.kaggle.com/alexisbcook/getting-started-with-halite)**!\n",
    "\n",
    "You can find more games as they're released on the **[Kaggle Simulations page](https://www.kaggle.com/simulations)**.\n",
    "\n",
    "As we did in the course, we recommend that you start simple, with an agent that follows your precise instructions.  This will allow you to learn more about the mechanics of the game and to build intuition for what makes a good agent.  Then, gradually increase the complexity of your agents to climb the leaderboard!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "papermill": {
     "duration": 0.370397,
     "end_time": "2020-10-01T00:42:36.705323",
     "exception": false,
     "start_time": "2020-10-01T00:42:36.334926",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161477) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "papermill": {
   "duration": 1010.557079,
   "end_time": "2020-10-01T00:42:37.142319",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2020-10-01T00:25:46.585240",
   "version": "2.1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}